Dynamics of a diffusive Leslie–Gower predator–prey system with ratio-dependent Holling III functional response

Abstract

This paper is devoted to investigating the dynamics of a diffusive Leslie–Gower predator–prey system with ratio-dependent Holling III functional response. We first establish the stability of positive constant equilibrium, and show the condition under which system undergoes a Hopf bifurcation with the explicit computational formulas for determining the bifurcating properties. Especially, when the positive constant equilibrium loses its stability, a supercritical Hopf bifurcation with spatial homogeneous and stable bifurcating periodic solution occurs. Finally, we discuss the existence and nonexistence of nonconstant positive solutions with the help of Leray–Schauder degree theory and the implicit function theorem.

1 Introduction

As one of the most common mutual relationships between two populations in nature, predator–prey relationship plays a significant role in ecology and mathematical biology. In mathematics, one way to model this relationship is by using differential equations with various types of predator’s functional responses, which is a key component of a predator–prey relationship. Leslie and Gower in [16, 17] introduced a functional response, called Leslie–Gower functional response, to describe that reduction in predator population has a reciprocal relationship with per capita availability of its preferred food. A Leslie–Gower predator–prey system generally has the following form:

$$ \textstyle\begin{cases} \frac{\mathrm{d}{x}}{\mathrm{d}{t}}=rx (1- \frac{x}{K} )-yf(x), \\ \frac{\mathrm{d}{y}}{\mathrm{d}{t}}=y (1- \frac{y}{px} ), \\ x(0)>0, y(0)>0, r,s,K,p>0, \end{cases} $$

(1.1)

where p is the conversion factor of the prey into the predator, and the term ${y}/{(px)}$ is called the Leslie–Gower term representing the loss of the density of the predator due to rarity of the prey.

In recent years, there have been many theoretical results obtained for system (1.1) with different functional responses $f(x)$, such as Holling type I ($f(x)=ax$) [3, 8, 12, 14], Holling type II (the so-called Holling–Tanner model, $f(x)=mx/(b+x)$) [2, 8, 9], Holling type III ($f(x)=mx^{2}/[(a+x)(b+x)]$) [8, 11], and Holling type IV ($f(x)=mx/(b+x^{2})$) [19]. There is a recognition that spatial distribution patterns and dispersal mechanisms can cause the complexity and rich dynamical behavior of ecological modeling. The Leslie–Gower predator–prey system with diffusion has been investigated in a certain range [4,5,6, 13, 22].

In characterizing predator’s functional response, ratio-dependent functional response is an important form. It is a predator-dependent functional response and has a better modeling for a predator–prey system [1, 10, 15, 18, 24, 25, 28, 32]. Motivated by the existing studies and the above considerations, we replace $f(x)$ by $f(x/y)$ and consider spatial distribution patterns and dispersal mechanisms in (1.1). Making a suitable nondimensional scaling, system (1.1) reduces to

$$ \textstyle\begin{cases} \frac{\partial {u}}{\partial {t}}=d_{1}\triangle {u} +u(1-u)- \frac{\alpha u^{2} v}{u^{2}+m v^{2}}, &x\in \varOmega , t>0, \\ \frac{\partial {v}}{\partial {t}}=d_{2}\triangle {v}+\beta v (1- \frac{v}{u} ), & x\in \varOmega , t>0, \\ \frac{\partial {u}}{\partial \nu }= \frac{\partial {v}}{\partial \nu }=0, & x\in \partial {\varOmega }, t \geq 0, \\ u(0,x)=u_{0}(x)\geq \not \equiv 0, v(0,x)=v_{0}(x)\geq \not \equiv 0,&x\in \overline{\varOmega }, \end{cases} $$

(1.2)

where $u(x,t)$ and $v(x,t)$ represent the densities of prey and predator at location $x\in \bar{\varOmega }$ and time $t\geq 0$, respectively. $\varOmega \in {\mathbb{R}}^{n}$ ($n\in {\mathbb{N}}$) is a bounded domain with smooth boundary ∂Ω. $d_{1}$ and $d_{2}$ are diffusion coefficients, and α, m, β are positive constants. In [26], Shi and Li introduced system (1.2) and established the stability of the positive equilibrium and uniform persistence of the solution semiflows. Shi et al. in [27] considered the existence of the Turing–Hopf bifurcation where the Turing instability curve and the Hopf bifurcation curve intersect. Yang and Li in [30] obtained a sufficient condition for global asymptotic stability of the positive equilibrium by constructing recurrent sequences and using an iterative method. Zhou in [33] established the existence and the bifurcating properties of Hopf bifurcation of system (1.2) by using β as a bifurcating value.

The purpose of the present paper is to investigate dynamics of system (1.2), including the stability of the unique positive equilibrium, the existence and the bifurcating properties of Hopf bifurcation, the existence and nonexistence of nonconstant positive solutions. The main contributions have three aspects:

• 1. A larger parameter range of local stability of the unique positive constant equilibrium is established;

• 2. Using the predation rate α as the bifurcating value, we show that system (1.2) undergoes a Hopf bifurcation and derive the explicit computational formulas for determining the bifurcating properties. In particular, a supercritical Hopf bifurcation with spatial homogeneous and stable bifurcating periodic solutions occurs;

• 3. If the diffusion coefficients $d_{1}$, $d_{2}$ are sufficiently large, then system (1.2) has no nonconstant positive solutions. With the help of Leray–Schauder degree theory, the existence of nonconstant positive solutions can be obtained when $d_{1}/d_{2}$ is small enough.

Our paper is organized as follows. In Sect. 2, we establish the stability of the unique positive constant equilibrium. In Sect. 3, we discuss the condition under which system (1.2) undergoes a Hopf bifurcation and derive the explicit computational formulas for determining the bifurcating properties. Especially, a supercritical Hopf bifurcation with spatial homogeneous and stable bifurcating periodic solutions occurs. Finally, we investigate the existence and nonexistence of nonconstant positive solutions by using a priori estimates, Leray–Schauder degree theory, and the implicit function theorem.

2 Stability of constant equilibria

In this section, we investigate the stability of constant equilibria of (1.2). From [26], we know that system (1.2) has a unique positive solution, denoted as $E^{*}=( \theta ,\theta )$ with $\theta =1-\alpha /(1+m)$, if $\alpha <1+m$. It is not difficult to see that system (1.2) has a semi-trivial constant solution $E_{1}=(1,0)$.

To discuss the stability of constant equilibria of system (1.2), we first make some notations. It is well known that the operator −Δ in Ω with the homogeneous Neumann boundary condition has eigenvalues

$$ \mu _{i}\in \varLambda := \{\mu _{i}:0=\mu _{0}< \mu _{1}< \cdots < \mu _{i}< \cdots , i\in {\mathbb{N}}_{0} \} $$

(2.1)

and the corresponding eigenfunctions are $\phi _{ij}$, where ${\mathbb{N}}_{0}:={\mathbb{N}}\cup \{0\}$. Let $S(\mu _{i})$ be the subspace generated by the eigenfunctions $\phi _{ij}$ corresponding to $\mu _{i}$, $m(\mu _{i})$ be the multiplicity of $\mu _{i}$, and $\{\phi _{ij}\}_{j=1}^{m(\mu _{i})}$ be an orthonormal basis of $S(\mu _{i})$. Define $X_{ij}=\{c\phi _{ij}:c\in {\mathbb{R}}^{2}\}$, $X_{i}=\bigoplus_{j=1}^{m(\mu _{i})}X_{ij}$, and

$$ {\mathbb{X}}= \bigl\{ (u,v)^{T}\in \bigl[C^{1}(\bar{\varOmega }) \bigr] ^{2}: {\partial _{\nu } u}={\partial _{\nu } v}=0 \text{ on } \partial \varOmega \bigr\} $$

satisfying ${\mathbb{X}}=\bigoplus_{i=0}^{\infty }X_{i}$.

The stability of the equilibria can be obtained by analyzing the distribution of the eigenvalues of the characteristic equation corresponding to the linearized system of system (1.2). Then we have the following results.

Theorem 2.1

$E_{1}=(1,0)$ is unstable.

Proof

The linearized system of system (1.2) at $E_{1}$ can be written as

$$\left(\begin{array}{c}{\phi }_t\\ {\psi }_t\end{array}\right)=D\mathrm{\Delta }\left(\begin{array}{c}\phi \\ \psi \end{array}\right)+\left(\begin{array}{cc}-1& -\alpha \\ 0& \beta \end{array}\right)\left(\begin{array}{c}\phi \\ \psi \end{array}\right),$$

(2.2)

with $D=\operatorname{diag}(d_{1},d_{2})$. By using the fact above, the corresponding kth characteristic equation of (2.2) is

$$ \lambda ^{2}-\bigl(-(d_{1}+d_{2})\mu _{k}+\beta -1\bigr)\lambda +\bigl(d_{1}d_{2}\mu _{k}^{2}-(\beta d_{1}-d_{2})\mu _{k}-\beta \bigr)=0. $$

(2.3)

When $k=0$, two roots of (2.3) are −1 and β, which implies that $E_{1}$ is unstable. □

Theorem 2.2

$E^{*}$ is locally asymptotically stable if one of the following statements holds:

(A):

$m \geq 1$ and $\alpha <1+m$;

(B):

$m<1$ and $\alpha \leq \frac{(1+m)^{2}}{2}$;

(C):

$m<1$, $\beta < \frac{1-m}{1+m}$, $\frac{(1+m)^{2}}{2}<\alpha <(1+\beta ) \frac{(1+m)^{2}}{2}$, and $\frac{d_{1}}{d_{2}}>\tilde{d}:= \frac{1-m}{\beta (1+m)}$;

(D):

$m<1$, $\beta \geq \frac{1-m}{1+m}$, $\frac{(1+m)^{2}}{2}<\alpha <1+m$, and $\frac{d_{1}}{d_{2}}>\tilde{d}$.

Proof

We now prove the local asymptotic stability of $E^{*}$ by analyzing the distribution of eigenvalues at $E^{*}$. The linearized system of system (1.2) at $E^{*}$ is

$$\left(\begin{array}{c}{\phi }_t\\ {\psi }_t\end{array}\right)=D\mathrm{\Delta }\left(\begin{array}{c}\phi \\ \psi \end{array}\right)+\left(\begin{array}{cc}{K}_1& K_2\\ \beta & -\beta \end{array}\right)\left(\begin{array}{c}\phi \\ \psi \end{array}\right),$$

(2.4)

and the kth characteristic equation of (2.4) is

$$ \lambda ^{2}-\mathrm{TR}_{k}(\alpha )\lambda +\mathrm{DET}_{k}(\alpha )=0, $$

(2.5)

where

$$ K_{1}= \frac{2\alpha }{(1+m)^{2}}-1, \qquad K_{2}= \frac{\alpha (m-1)}{(1+m)^{2}} $$

(2.6)

and

$$ \begin{aligned}[b] &\mathrm{TR}_{k}(\alpha )=-(d_{1}+d_{2})\mu _{k}+K_{1}-\beta , \\ &\mathrm{DET}_{k}(\alpha )=d_{1}d_{2}\mu _{k}^{2}- (K_{1}d_{2}-\beta d _{1} )\mu _{k}+\beta \theta . \end{aligned} $$

(2.7)

Note that if $\mathrm{TR}_{i}(\alpha )<0$ and $\mathrm{DET}_{j}(\alpha )>0$ for all $i,j\in {\mathbb{N}}_{0}$, then $E^{*}$ is locally asymptotically stable. It is clear that if $K_{1}<\beta$, then $\mathrm{TR}_{0}(\alpha )<0$ and $\mathrm{TR}_{k}(\alpha )<0$ for all $k\in {\mathbb{N}}$. If $K_{1}\leq 0$, that is, $\alpha \leq (1+m)^{2}/2$, then, obviously, $\mathrm{TR}_{k}(\alpha )<0$ and $\mathrm{DET}_{k}(\alpha )>0$ for all $k\in {\mathbb{N}}_{0}$. If $0< K_{1}< \beta$, that is, $(1+m)^{2}/2<\alpha <(1+\beta )(1+m)^{2}/2$, then $\mathrm{TR}_{0}(\alpha )<0$ and $\mathrm{TR}_{j}(\alpha _{0})<0$ for all $j\in {\mathbb{N}}$. Note that

$$\begin{aligned} \mathrm{DET}_{k}(\alpha )&=d_{1}d_{2}\mu _{k}^{2}- \biggl( \biggl( \frac{2\alpha }{(1+m)^{2}}-1 \biggr)d_{2}-\beta d_{1} \biggr)\mu _{k}+ \beta \theta \\ &> d_{1}d_{2}\mu _{k}^{2}- \biggl( \biggl( \frac{2 (1+m)}{(1+m)^{2}}-1 \biggr)d_{2}-\beta d_{1} \biggr) \mu _{k} \\ &=\mu _{k} \biggl(d_{1}d_{2}\mu _{k}- \biggl( \biggl( \frac{1-m }{(1+m)} \biggr)d_{2}-\beta d_{1} \biggr) \biggr). \end{aligned}$$

Then $\mathrm{DET}_{k}(\alpha )>0$ for all $\alpha <1+m$, if $m<1$ and $d_{1}/d_{2}>(1-m)/(\beta (1+m))$ or $m\geq 1$. To sum up above, we obtain the local stability of $E^{*}$ and complete the proof. □

Figure 1 shows the region of stability of equilibrium $E^{*}$ in the $m-\alpha$ plane. $E^{*}$ is globally stable with the $m-\alpha$ plane belonging to the yellow region, which is given as $(\varepsilon _{0}+\theta )(1+m)^{2}\varepsilon _{0}^{2}-\alpha (1-m \varepsilon _{0})(1+\theta )>0$ with $\varepsilon _{0}=1-\alpha /(2 \sqrt{m})>0$ in [26]. $E^{*}$ is locally asymptotically stable with the $m-\alpha$ plane belonging to the red region (corresponding to case (A) and case (B) in Theorem 2.2), which is also given in [26]. Here we give a new local stability region of $E^{*}$ (see the blue region) corresponding to case (C) or case (D) of Theorem 2.2.

Figure 1

The parameter ranges in the $m-\alpha$ plane for the stability region of $E^{*}$. Here $d_{1}=3$, $d_{2}=0.5$

3 Hopf bifurcation and its properites

In this section, we focus on the case that system (1.2) undergoes a Hopf bifurcation near $E^{*}$ for the spatial domain $\varOmega =(0,l\pi )$ with $l\in {\mathbb{R}}^{+}=(0,+\infty )$. Then the eigenvalue $\mu _{n}$ is $n^{2}/l^{2}$ and the corresponding eigenfunction $\phi _{n}(x)$ is $\cos (nx/l)$ for all $n\in {\mathbb{N}} _{0}$ and $x\in \varOmega$. Our results in this section can also be adapted to higher spatial domains. By cases (A) and (B) of Theorem 2.2, the potential bifurcating value should satisfy $\alpha \in ( (1+m)^{2}/2, 1+m)$ with $m<1$. To study Hopf bifurcation near $E^{*}$, we need to verify the condition under which the corresponding characteristic equations of the linearized operator of system (1.2) have a pair of simply pure imaginary roots, all other eigenvalues have non-zero real parts, and the transversality condition holds.

From [31], if $\alpha ^{*}$ is a Hopf bifurcation value, then there exists $k\in {\mathbb{N}}_{0}$ such that

$$ \begin{gathered} \mathrm{TR}_{k}\bigl(\alpha ^{*}\bigr)=0, \qquad \mathrm{DET}_{k}\bigl(\alpha ^{*}\bigr)>0,\quad \text{and}\\ \mathrm{TR}_{j}\bigl( \alpha ^{*}\bigr)\neq 0,\qquad \mathrm{DET}_{j}\bigl(\alpha ^{*}\bigr)\neq 0,\quad \text{for } j\in {\mathbb{N}} _{0}/\{k\}, \end{gathered} $$

(3.1)

and for the unique pair of complex eigenvalues $\delta (\alpha ) \pm i\omega (\alpha )$ near the imaginary axis,

$$ \delta '\bigl(\alpha ^{*}\bigr)\neq 0, $$

(3.2)

where $\mathrm{TR}_{k}(\alpha )$ and $\mathrm{DET}_{k}(\alpha )$ are defined in (2.7).

If $k=0$, then $\mathrm{TR}_{0}(\alpha )=K_{1}-\beta$, $\mathrm{DET}_{0}(\alpha )=\beta \theta >0$. Let $\mathrm{TR}_{0}(\alpha )=0$, then $\alpha =\alpha _{0}:=(1+ \beta )(1+m)^{2}/2$ with $\beta <(1-m)/(1+m)$, and $\mathrm{TR}_{j}(\alpha _{0})<0$, $\mathrm{DET}_{j}(\alpha _{0})>0$ if $d_{1}/d_{2}> \tilde{d}$ for all $j\in {\mathbb{N}}$, which implies that there exists a pair of simple pure imaginary roots of the characteristic equations of system (2.5) at $E^{*}$, and other eigenvalues has negative real parts.

We now consider $\alpha _{0}<\alpha <1+m$. It follows from $\mathrm{TR}_{k}( \alpha )=0$ that $\alpha =\alpha _{k}:=\alpha _{0}+(1+m)^{2}(d_{1}+d _{2})k^{2}/(2l^{2})$. In this case, for each $j\in {\mathbb{N}}_{0}/ \{k\}$, $\mathrm{TR}_{j}(\alpha _{k})\neq 0$, $\mathrm{DET}_{j}(\alpha _{k})>0$ if $d_{1}/d_{2}>\tilde{d}$. It is noted that there are only finite positive integers k satisfying (3.1) since $\alpha _{k}\in (\alpha _{0}, 1+m)$. Then the maximum positive integer, denoted as $N_{1}$, is the integer part of $l\sqrt{(1-m-\beta (1+m))/((d_{1}+d_{2})(1+m))}$.

We next verify the transversality condition (3.2). If $\delta (\alpha )\pm i \omega (\alpha )$ are the roots of (2.5), then

$$ \delta (\alpha )= \frac{\mathrm{TR}_{k}(\alpha )}{2}=- \frac{(d_{1}+d_{2})}{2} \frac{k^{2}}{l^{2}}+ \frac{\alpha }{(1+m)^{2}}- \frac{1+\beta }{2}, \qquad \omega (\alpha )=\sqrt{ \mathrm{DET}_{k}(\alpha )-\delta ^{2}(\alpha )}, $$

and

$$ \delta '\bigl(\alpha ^{*}\bigr)= \frac{1}{(1+m)^{2}}>0, $$

where $\alpha ^{*}$ represents one of the bifurcating values $\alpha _{k}$ with $k\in [0, N_{1}]$.

Summarizing our analysis above, we obtain the following conclusion.

Theorem 3.1

Assume that $m<1$ and $\beta <(1-m)/(1+m)$. If ${d_{1}}/{d_{2}}> \tilde{d}$, then system (1.2) undergoes a Hopf bifurcation near $E^{*}$ at $\alpha =\alpha _{k}$, where

$$ \alpha _{k}= \frac{(1+m)^{2}}{2} \biggl(1+\beta +(d_{1}+d_{2}) \frac{k^{2}}{l^{2}} \biggr),\quad k\in [0, N_{1}], $$

and $N_{1}$ is the integer part of $l\sqrt{(1-m-\beta (1+m))/((d _{1}+d_{2})(1+m))}$. Moreover, the spatial homogeneous bifurcating periodic solutions bifurcate from $\alpha =\alpha _{0}$, and the spatial non-homogeneous bifurcating periodic solutions bifurcate from $\alpha =\alpha _{k}$ with $k\in [1, N_{1}]$.

We now establish the computational formulas for determining the properties of Hopf bifurcation, including the bifurcating direction and stability of periodic solutions bifurcating from $E^{*}$ at $\alpha =\alpha _{k}$ with $k\in [0,N_{1}]$, by using the normal form theory and the center manifold argument presented in [7, 31]. Fix $k\in [0, N_{1}]$, let $\tilde{\alpha }=\alpha _{k}$, $\tilde{\omega }=\omega (\alpha _{k})$, $\tilde{\theta }=1-\tilde{\alpha }/(1+m)$, and $\lambda =\pm i\omega ( \alpha _{k})=\pm i\tilde{\omega }$ be the purely imaginary roots. By making the change of variables $u(x,t)-\tilde{\theta }\longrightarrow u(x,t)$ and $v(x,t)-\tilde{\theta }\longrightarrow v(x,t)$, system (1.2) can be rewritten as an abstract form at $\alpha = \tilde{\alpha }$ by

$$ \dot{U}=L(\tilde{\alpha })U+F(\tilde{\alpha },U), $$

(3.3)

where $U=(u,v)^{T}$,

$$\begin{array}{rl}& L(\tilde{\alpha })=\left(\begin{array}{cc}{d}_1& 0\\ 0& d_2\end{array}\right)\mathrm{\Delta }+\left(\begin{array}{cc}{\tilde{K}}_1& {\tilde{K}}_2\\ \beta & -\beta \end{array}\right)\\ & \phantom{L(\tilde{\alpha })}=\left(\begin{array}{cc}{d}_1& 0\\ 0& d_2\end{array}\right)\mathrm{\Delta }+\left(\begin{array}{cc}\frac{2\tilde{\alpha }}{{(1+m)}^2}-1& \frac{\tilde{\alpha }(m-1)}{{(1+m)}^2}\\ \beta & -\beta \end{array}\right),\\ & F(\tilde{\alpha },U)=\left(\begin{array}{c}{F}_1(\tilde{\alpha },U)\\ F_2(\tilde{\alpha },U)\end{array}\right),\end{array}$$

(3.4)

and

$$\begin{aligned} &F_{1}(\tilde{\alpha }, U)= \frac{A_{1}}{(1 + m)^{2} ((1 + m) \tilde{\theta }^{2} + u^{2} + m v ^{2} + 2 \tilde{\theta }(u + m v))}, \\ &F_{2}(\tilde{\alpha }, U)=\beta (v+\tilde{\theta }) \biggl(1- \frac{v+\tilde{\theta }}{u+\tilde{\theta }} \biggr)-\beta u+\beta v- \frac{\beta (u-v)^{2}}{u+\tilde{\theta }}, \\ &A_{1}= -(1 + m)^{3} \tilde{\theta }^{4} + \bigl( 2 (1 + m)^{2}-2 \tilde{\alpha }\bigr) u^{3} - (1 + m)^{2}u^{4} + 2 m \bigl( (1 + m)^{2}- \tilde{ \alpha }\bigr) u v^{2} \\ &\hphantom{A_{1}=}{}- \tilde{\alpha }( m-1) m v^{3}- m u^{2} v \bigl(\tilde{\alpha }(3 + m) + (1 + m)^{2} v \bigr) - (1 + m)^{2} \tilde{\theta }^{3} \bigl( \tilde{\alpha }-1 + 4 u \\ &\hphantom{A_{1}=}{}+ m (2 u + 2 v-1 )\bigr) - \tilde{\theta }\bigl(4 (1 + m)^{2} u^{3}+ 2 m u \bigl(\tilde{\alpha }(5 + m)+ (1 + m)^{2} ( v-2)\bigr) v \\ &\hphantom{A_{1}=}{}- m \bigl(2 \tilde{\alpha }(1 - m) + (1 + m)^{2}\bigr) v^{2} + u^{2} \bigl(\tilde{\alpha }\bigl(5 + 2 m + m^{2}\bigr)+ (1 + m)^{2} ( 2 m v-5)\bigr)\bigr) \\ &\hphantom{A_{1}=}{} - (1 +m) \tilde{\theta }^{2} \bigl(\bigl(6 + 7 m + m ^{2}\bigr) u^{2} + m \bigl(2 \tilde{\alpha }+ (1 + m) (-2 + v)\bigr) v \\ &\hphantom{A_{1}=}{}+2 u \bigl(\tilde{\alpha }(2 + m)+ (1 + m) (2 m v-2 - m )\bigr)\bigr). \end{aligned}$$

Define $\langle \cdot , \cdot \rangle$ to be the complex-valued $L^{2}$ inner product on a Hilbert space ${\mathbb{X}}_{{\mathbb{C}}}$ as

$$ \langle U_{1}, U_{2}\rangle = \int ^{l\pi }_{0} (\bar{u}_{1} u_{2}+ \bar{v}_{1} v_{2})\,\mathrm{d}x, \quad U_{i}=(u_{i}, v_{i})^{T}\in { \mathbb{X}}_{{\mathbb{C}}}:={\mathbb{X}}+i {\mathbb{X}} \text{ with }i=1,2. $$

Let

$$L^{\ast }(\tilde{\alpha })=\left(\begin{array}{cc}{d}_1\mathrm{\Delta }+{\tilde{K}}_1& \beta \\ {\tilde{K}}_2& d_2\mathrm{\Delta }-\beta \end{array}\right).$$

Then the corresponding eigenfunctions of $L(\tilde{\alpha })$ and $L^{*}(\tilde{\alpha })$ for eigenvalues $\lambda =\pm i \tilde{\omega }$ are

$$ q=q(x)=(1,b_{k})^{T}\cos (kx/l), \qquad q^{*}=q^{*}(x)=\bigl(a_{k}^{*},b_{k}^{*} \bigr)^{T}\cos (kx/l), $$

respectively, where

$$ \begin{gathered} b_{k}= \frac{d_{1}\mu _{k}-\tilde{K}_{1}}{\tilde{K}_{2}}+ \frac{\tilde{\omega }}{\tilde{K}_{2}}i, \qquad a_{k}^{*}= \frac{1}{2\varGamma _{k}}+ \frac{\tilde{K}_{1}-d_{1}\mu _{k}}{2\tilde{\omega }\varGamma _{k} }i, \\ b_{k}^{*}= \frac{\tilde{K}_{2}}{2\tilde{\omega }\varGamma _{k}}i, \qquad \varGamma _{k}= \int ^{l\pi }_{0}\cos ^{2} \biggl( \frac{k}{l}x \biggr)\,dx, \end{gathered} $$

and q and $q^{*}$ satisfy $\langle q^{*},q\rangle =1$ and $\langle q^{*},\bar{q}\rangle =0$.

Let ${\mathbb{X}}={\mathbb{X}}_{C} \oplus {\mathbb{X}}_{S}$ with

$$ {\mathbb{X}}_{C}= \{zq+\bar{z}\bar{q}: z\in {\mathbb{C}} \}, \qquad { \mathbb{X}}_{S}= \bigl\{ U\in {\mathbb{X}}: \bigl\langle q^{*},U\bigr\rangle =0 \bigr\} . $$

Thus, for any $U=(u,v)^{T}\in {\mathbb{X}}$, there exist $z\in {\mathbb{C}}$ and $w=(w_{1},w_{2})^{T}\in {\mathbb{X}}_{S}$ such that $U=zq+\bar{z}\bar{q}+w$. Substituting it into system (3.3), we get

$$ \textstyle\begin{cases} \dot{z}=i \tilde{\omega }z+\langle q^{*}, F_{0}\rangle , \\ \dot{w}=L(\tilde{\alpha })w+H(z,\bar{z}, w), \end{cases} $$

(3.5)

where

$$ H(z,\bar{z}, w)=F_{0}-\bigl\langle q^{*},F_{0}\bigr\rangle q -\bigl\langle \bar{q} ^{*},F_{0}\bigr\rangle \bar{q}, $$

(3.6)

and

$$ F_{0}=F(\tilde{\alpha }, zq+\bar{z}\bar{q}+w) = \frac{1}{2}Q_{UU}+ \frac{1}{6}C_{UUU}+\mathcal{O}\bigl( \vert U \vert ^{4}\bigr), $$

here,

$$ Q_{qq}=(c_{k}, d_{k})^{T}\cos ^{2}(kx/l),\qquad Q_{q\bar{q}}=(e_{k}, f_{k})^{T} \cos ^{2}(kx/l),\qquad C_{qq\bar{q}}=(g_{k}, h_{k})^{T}\cos ^{3}(kx/l) $$

and

$$\begin{aligned} &c_{k}= -2 + \frac{2 \tilde{\alpha }( m-3) m (\tilde{\omega }^{2}-(\tilde{\theta }+ d_{1} \mu _{k})^{2} )}{ \tilde{K}_{2}^{2} (1 + m)^{3} \tilde{\theta }} + i \frac{4 \tilde{\alpha }m(3-m) (\tilde{\theta }+ d_{1} \mu _{k} ) \tilde{\omega }}{\tilde{K}_{2}^{2} (1 + m)^{3} \tilde{\theta }}, \\ &d_{k}= \frac{2 \beta (\tilde{\omega }^{2}-(\tilde{\theta }+ d_{1} \mu _{k})^{2} )}{\tilde{K}_{2}^{2} \tilde{\theta }}-i \frac{4 \beta (\tilde{\theta }+ d_{1} \mu _{k}) \tilde{\omega }}{ \tilde{K}_{2}^{2} \tilde{\theta }}, \\ &e_{k}= \frac{ 2 \tilde{\alpha }( 3-m) m ((\tilde{\theta }+ d_{1} \mu _{k})^{2} + \tilde{\omega }^{2})}{ \tilde{K}_{2}^{2} (1 + m)^{3} \tilde{\theta }}-2, \\ &f_{k}=- \frac{2 \beta ( (\tilde{\theta }+ d_{1} \mu _{k})^{2} + \tilde{\omega }^{2})}{\tilde{K}_{2}^{2} \tilde{\theta }}, \\ &g_{k}= \frac{-2 \tilde{\alpha }m}{\tilde{K}_{2}^{3} (1 + m)^{4} \tilde{\theta }^{2}}\bigl(3 (\tilde{\theta }+ d_{1} \mu _{k})^{2} \bigl(4 \tilde{K}_{2} ( 1-m) + \tilde{K}_{1} \bigl(1 + ( m-6) m\bigr) \\ &\hphantom{g_{k}=}{}- d_{1} \bigl(1 + (m-6) m\bigr) \mu _{k}\bigr) + \bigl(2 \tilde{K}_{2} \bigl(3+ (m-8) m\bigr) + 3 \tilde{K}_{1} \bigl(1 + ( m-6) m\bigr)\\ &\hphantom{g_{k}=}{}- 3 d_{1} \bigl(1 + ( m-6) m\bigr) \mu _{k}\bigr) \tilde{\omega }^{2}\bigr) + \frac{i2 \tilde{\alpha }m \tilde{\omega }}{\tilde{K}_{2}^{3} (1 + m)^{4} \tilde{\theta }^{2}} \bigl(\tilde{K}_{2}^{2} \bigl(9+ m^{2}-14 m\bigr) \\ &\hphantom{g_{k}=}{} + 4 \tilde{K}_{1} \tilde{K}_{2} \bigl(3 + (m-8) m\bigr) + 3 \tilde{K} _{1}^{2} \bigl(1 + ( m-6) m\bigr) -4 d_{1} \tilde{K}_{2} \bigl(3 + (m-8) m\bigr) \mu _{k} \\ &\hphantom{g_{k}=}{}- 6 d_{1} \tilde{K}_{1} \bigl(1+ ( m-6) m\bigr) \mu _{k} + 3 \bigl(1 + (m-6) m\bigr) \bigl(d_{1}^{2} \mu _{k}^{2} + \tilde{ \omega }^{2}\bigr)\bigr) , \\ &h_{k}= \frac{2 \beta (3 (\tilde{\theta }+ d_{1} \mu _{k})^{2} + \tilde{\omega }^{2})}{\tilde{K}_{2}^{2} \tilde{\theta }^{2}}+ \frac{4 \beta (\tilde{\theta }+ d_{1} \mu _{k}) \tilde{\omega }}{ \tilde{K}_{2}^{2} \tilde{\theta }^{2}}i. \end{aligned}$$

Denote

$$ H(z,\bar{z}, w)= \frac{H_{20}}{2}z^{2}+H_{11}z \bar{z}+ \frac{H_{02}}{2}\bar{z} ^{2}+ \mathcal{O}\bigl( \vert z \vert \cdot \vert w \vert \bigr). $$

(3.7)

Then comparing the coefficients of the same order of z and z̄ in (3.6) and (3.7), we have

$$\begin{aligned} H_{20}&= \textstyle\begin{cases} 0, &k=0, \\ ( c_{k}\ d_{k} )^{T} \cos ^{2} ( \frac{k}{l}x ), &k\in {\mathbb{N}}, \end{cases}\displaystyle \qquad H_{11}= \textstyle\begin{cases} 0, &k=0, \\ (e_{k} \ f_{k} )^{T} \cos ^{2} ( \frac{k}{l}x ), &k\in {\mathbb{N}}. \end{cases}\displaystyle \end{aligned}$$

Let

$$ w= \frac{w_{20}}{2}z^{2}+w_{11}z\bar{z}+ \frac{w_{02}}{2}\bar{z}^{2}+\mathcal{O}\bigl( \vert z \vert ^{3}\bigr). $$

Then, by [7, 31], we have

$$\begin{array}{c}{w}_{20}=\{\begin{array}{cc}\frac{-1}{3{\tilde{\omega }}^2}\left(\begin{array}{c}(\beta +d_2{\mu }_k+2\tilde{\omega }i)c_k+{\tilde{K}}_2{d}_k\\ \beta c_k+(d_1{\mu }_k-{\tilde{K}}_1+2\tilde{\omega }i)d_k\end{array}\right){cos}^2(\frac{k}{l}x),\hfill & k\in \mathbb{N},\hfill \\ 0,\hfill & k=0,\hfill \end{array}\hfill \\ w_{11}=\{\begin{array}{cc}\frac{1}{{\tilde{\omega }}^2}\left(\begin{array}{c}(\beta +d_2{\mu }_k)e_k+{\tilde{K}}_2{f}_k\\ \beta e_k+(d_1{\mu }_k-{\tilde{K}}_1)f_k\end{array}\right){cos}^2(\frac{k}{l}x),\hfill & k\in \mathbb{N},\hfill \\ 0,\hfill & k=0.\hfill \end{array}\hfill \end{array}$$

Denote

$$ \dot{z}=i\tilde{\omega }z+\sum_{2\leq i+j \leq 3} \frac{g_{ij}}{i!j!}z^{i}\bar{z}^{j}+\mathcal{O}\bigl( \vert z \vert ^{4}\bigr). $$

(3.8)

Then, comparing (3.5) and (3.8), when $k=0$, we have $Q_{w_{11}q}=Q_{w_{20}\bar{q}}=0$,

$$\begin{aligned} &g_{20}= \frac{1}{\tilde{K}_{2}^{2} (1 + m)^{3} \tilde{\theta }} \bigl( \tilde{\alpha }( m-3) m \bigl( \tilde{K}_{1}^{2} - \tilde{K}_{2}^{2} + \tilde{\omega }^{2}\bigr)-2 \beta \tilde{K}_{2} \tilde{\theta }(1 + m)^{3} - \tilde{K}_{2}^{2} (1 + m)^{3} \tilde{\theta }\bigr) \\ &\hphantom{g_{20}=}+ \frac{i}{\tilde{K}_{2}^{2} (1 + m)^{3} \tilde{\omega }\tilde{\theta }} \bigl(\beta\tilde{K}_{2} (1 + m)^{3} \bigl(\tilde{\theta }^{2} - \tilde{\omega }^{2}\bigr)\bigr) + \tilde{\alpha }m(m-3) \bigl(\tilde{K}_{1} \tilde{\theta }^{2} + ( \tilde{K}_{1} + 2 \tilde{K}_{2}) \tilde{\omega }^{2} \bigr) \\ &\hphantom{g_{20}=}+ \tilde{K}_{1} \tilde{K}_{2}^{2} (1 + m)^{3} \tilde{\theta }), \\ &g_{11}= -1 - \frac{\tilde{\alpha }(m-3) m (\tilde{\theta }^{2} +\tilde{\omega } ^{2})}{\tilde{K}_{2}^{2} (1 + m)^{3}\tilde{\theta }} + \frac{i}{\tilde{K}_{2}^{2} (1 + m)^{3} \tilde{\omega }\tilde{\theta }}\bigl(\bigl( \tilde{\theta }^{2} + \tilde{\omega }^{2}\bigr) \bigl(\tilde{ \alpha }m \tilde{K}_{1} ( m-3) \\ &\hphantom{g_{11}=} + \beta \tilde{K}_{2}(1+ m)^{3}\bigr) +\tilde{K}_{1} \tilde{K}_{2}^{2}(1 + m)^{3} \tilde{\theta }\bigr), \\ &g_{21}= \frac{8 (3 ( 1-m) m + \beta (1 +5 m - 2 m^{2}))}{(1 + \beta ) (1 + m)^{2} (\beta (1-m^{2}) - (m-1)^{2} )} +i \frac{2 \beta (\beta (1+m+4m^{2})+(4m^{2}-3m-3))}{(1 + \beta ) ( \beta (1+m) + m-1 ) ( 1- m^{2}) \tilde{\omega })}, \end{aligned}$$

and when $k\in {\mathbb{N}}$, we get $g_{20}=g_{11}=g_{02}=0$ and

$$\begin{aligned} g_{21}&=2\bigl\langle q^{*},Q_{w_{11}q}\bigr\rangle + \bigl\langle q^{*},Q_{w_{20} \bar{q}}\bigr\rangle + \bigl\langle q^{*},C_{qq\bar{q}}\bigr\rangle =g_{21R}+ig_{21I}, \end{aligned}$$

where

$$\begin{aligned} &g_{21R}= \frac{-1}{2 \tilde{K}_{2}^{3} (1 + m)^{6}\tilde{\omega }^{2} \tilde{\theta }^{2}} (-3 \beta ^{2} \tilde{K}_{2} (1 + m)^{6} ( \tilde{\theta }+ d_{1} \mu _{k}) \bigl((\tilde{\theta }+ d_{1}\mu _{k})^{2} + \tilde{\omega }^{2}\bigr) \\ &\hphantom{g_{21R}=}{}+ \tilde{ \alpha }^{2}( m-3)^{2} m^{2} \bigl(( \tilde{\theta }+ d_{1} \mu _{k})^{2}+ \tilde{\omega }^{2}\bigr)\bigl( 2 d_{1}^{2} \mu _{k}^{2} + 2 \tilde{\omega }^{2}-2 \tilde{K}_{1} \tilde{\theta }\\ &\hphantom{g_{21R}=}{} - \bigl(5 d_{2} \tilde{K}_{2}+ 2 d_{1} (2 \tilde{K}_{1}+ \tilde{K}_{2})\bigr)\mu _{k}\bigr) + \tilde{K}_{2}^{3}(2 \tilde{K}_{1}- 2 d_{1} \mu _{k} - 5 d_{2} \mu _{k}) \tilde{\theta }^{2}(1+ m)^{6} \\ &\hphantom{g_{21R}=}{}+\alpha \tilde{K}_{2} m (1 + m)^{2}(3 \tilde{K}_{1} \bigl(\tilde{K}_{2} \bigl(9+ m^{2}-14 m\bigr) - 2 d _{1} \bigl(3 + m^{2}-8 m\bigr) \mu _{k}\bigr) \tilde{\omega }^{2} \\ &\hphantom{g_{21R}=}{}+ 3 \tilde{\omega } ^{2} \bigl(-6\tilde{K}_{2}^{2} ( m-1) - d_{1} \tilde{K}_{2}\bigl(9 + m^{2} -14 m\bigr) \mu _{k} + \bigl(3 + ( m -8) m\bigr) \bigl(d_{1}^{2} \mu _{k}^{2} + \tilde{ \omega }^{2}\bigr)\bigr)\\ &\hphantom{g_{21R}=}{}+ 2 \tilde{K}_{1} ( m -3) (1 + m) \bigl(\tilde{K}_{1} ^{2}+2 \tilde{K}_{2}^{2}- (6 d_{1}+ 5 d_{2}) \tilde{K}_{2} \mu _{k} + d _{1}\mu _{k}^{2}(3 d_{1} + 7 d_{2}) + \tilde{\omega }^{2}\bigr) \tilde{\theta } \\ &\hphantom{g_{21R}=}{}- ( m-3) (1 + m) \bigl(\mu _{k} \bigl(2 (2 d_{1} + 5 d_{2}) \tilde{K}_{2}^{2} -2 d_{1} (3 d_{1} + 5 d_{2})\tilde{K}_{2} \mu _{k} \\ &\hphantom{g_{21R}=}{}+ d_{1}^{2} (2 d_{1} + 7 d_{2}) \mu _{k}^{2}\bigr)+ ( 2 d_{1} \mu _{k} -2 \tilde{K}_{2}+ 7 d_{2} \mu _{k}) \tilde{\omega }^{2}\bigr) \tilde{\theta } \\ &\hphantom{g_{21R}=}{}+ \tilde{K}_{1}^{2} \bigl(3 \bigl(3 + (m-8) m\bigr) \tilde{ \omega }^{2}+ ( m-3) (1 + m) \tilde{\theta }(6 \tilde{K} _{2} - 6 d_{1} \mu _{k} - 7 d_{2} \mu _{k})\bigr))\\ &\hphantom{g_{21R}=}{} - \beta (1 + m)^{3} \bigl(\tilde{\alpha }( m-3 ) m \bigl(\bigl(( \tilde{\theta }+ d_{1} \mu _{k})^{2} + \tilde{\omega }^{2}\bigr) \bigl(2 \tilde{K}_{1}^{2} + 3 \tilde{K}_{2}^{2} - 5 (d_{1} + d_{2}) \tilde{K} _{2} \mu _{k} \\ &\hphantom{g_{21R}=}{}+ 2 d_{1}^{2} \mu _{k}^{2} + \tilde{K}_{1} (5 \tilde{K} _{2}- 4 d_{1} \mu _{k}) + 2 \tilde{\omega }^{2}\bigr) + \tilde{K}_{2} \bigl(7 \tilde{\omega }^{2}-7 \tilde{K}_{1}-5 \tilde{K}_{2}\\ &\hphantom{g_{21R}=}{} +7 d_{1} \mu _{k}\bigr) ( \tilde{\theta }+ d_{1} \mu _{k}) \bigr) \tilde{\theta }\bigr) + \tilde{K}_{2}^{2} (1 + m)^{3} \bigl(-3 (-\tilde{\theta }- d_{1} \mu _{k}) \tilde{\omega }^{2}+\bigl(7 \tilde{K}_{1}^{2} + 10 \tilde{K}_{1} \tilde{K}_{2} \\ &\hphantom{g_{21R}=}{} + 3\tilde{K} _{2}^{2}- \bigl(7 d_{2} \tilde{K}_{2} + 2 d_{1} (7 \tilde{K}_{1} + 5 \tilde{K}_{2})\bigr) \mu _{k} +7 d_{1}^{2} \mu _{k}^{2} + 7 \tilde{\omega } ^{2}\bigr) \tilde{\theta }+ 5 \tilde{K}_{2} \tilde{\theta }^{2}\bigr))), \\ &g_{21I}= \frac{-1}{4 \tilde{K}_{2}^{3} (1 + m)^{6} \tilde{\omega }^{3} \tilde{\theta }^{2}}(2 \beta ^{2} \tilde{K}_{2} (1 + m)^{6} \bigl(5 ( \tilde{\theta }+ d_{1} \mu _{k})^{4} + 7(\tilde{\theta }+ d_{1} \mu _{k})^{2} \tilde{\omega }^{2} + 2\tilde{\omega }^{4}\bigr) \\ &\hphantom{g_{21I}=}{}+ 2 \tilde{\alpha }^{2} m^{2}(m-3)^{2}\bigl((\tilde{\theta }+d_{1}\mu _{k})^{2} +\tilde{\omega }^{2}\bigr) \bigl(2 \tilde{K}_{2}\tilde{\omega } ^{2} + 5 d_{2} \mu _{k} \bigl((d_{1} \mu _{k}-\tilde{K}_{1} ) ( \tilde{\theta }+ d_{1} \mu _{k})\\ &\hphantom{g_{21I}=}{} + \tilde{\omega }^{2}\bigr)\bigr) + 2 \tilde{K} _{2}^{3}\bigl(2 \tilde{\omega }^{2} +5 d_{2} \mu _{k} ( \tilde{K}_{1}- d_{1} \mu _{k}) \bigr) (1+ m)^{6}\tilde{\theta }^{2}- \tilde{\alpha }m (1 + m)^{2} (9 \tilde{K}_{1} ^{4} \bigl(1 \\ &\hphantom{g_{21I}=}{} + (m-6) m\bigr) \tilde{\omega }^{2} + 9 d_{1}^{4} \bigl(1 +m( m-6)\bigr) \mu _{k}^{4} \tilde{\omega }^{2} + 3 \tilde{K}_{2}^{2} \bigl(9+ ( m-14) m\bigr) \tilde{\omega }^{4} \\ &\hphantom{g_{21I}=}{}+ 9\bigl(1+m^{2}-6 m\bigr) \tilde{\omega }^{6} - 2 \tilde{K}_{2} ( m-3) (1 + m) \tilde{\omega }^{2}\bigl( 5 d_{2} \tilde{K}_{2} \mu _{k}+4 \tilde{K}_{2} ^{2}- 2 \tilde{\omega }^{2}\bigr) \tilde{\theta } \\ &\hphantom{g_{21I}=}{}+ 2 d_{1}^{3} \tilde{K} _{2} \mu _{k}^{3} \bigl(-9 \bigl(3 + m^{2}-8 m\bigr) \tilde{\omega }^{2}+ 5 d_{2} ( m-3) (1 + m) \mu _{k} \tilde{\theta }\bigr)+ 2 \tilde{K}_{1} ^{3} \bigl(9 \bigl(\tilde{K}_{2} \bigl(3+ m^{2}\\ &\hphantom{g_{21I}=}{}-8 m\bigr) - 2 d_{1} \bigl(1 + (m-6) m\bigr) \mu _{k}\bigr) \tilde{\omega }^{2} - 5 d_{2} \tilde{K}_{2} \tilde{\theta }( m-3)(1 + m) \mu _{k}\bigr)+ d_{1}^{2}\mu _{k}^{2}\tilde{\omega }^{2} \bigl(9 \tilde{K}_{2}^{2} \bigl(9 \\ &\hphantom{g_{21I}=}{} + m^{2}-14 m\bigr)+ 18 \bigl(1+ m^{2}-6 m\bigr) \tilde{\omega } ^{4} - 2 \tilde{K}_{2} ( m-3) (1 + m)\tilde{\theta }\bigl(15 d_{2} \tilde{K}_{2} \mu _{k}- 2 \tilde{\omega }^{2}\bigr)\bigr) \\ &\hphantom{g_{21I}=}{}+ \tilde{K}_{1}^{2} \bigl(9 \tilde{\omega }^{2} \bigl( \tilde{K}_{2}^{2} \bigl(9 + m^{2}-14 m\bigr) - 6 d_{1} \tilde{K}_{2} \bigl(3 + m^{2}-8 m\bigr) \mu _{k} + 2 \bigl(1 + m^{2} \\ &\hphantom{g_{21I}=}{}-6 m\bigr) \bigl(3 d_{1}^{2} \mu _{k}^{2} + \tilde{\omega }^{2}\bigr)\bigr)- 2 \tilde{K}_{2}\bigl(m^{2}-2m-3\bigr) \bigl(15 d_{2} \mu _{k} (\tilde{K}_{2} - d _{1} \mu _{k}) - 2 \tilde{\omega }^{2}\bigr) \tilde{\theta }\bigr) \\ &\hphantom{g_{21I}=}{}+ 2 d_{1} \tilde{K}_{2} \mu _{k} \bigl(18\tilde{ \omega }^{2} \tilde{K}_{2}^{2}(m-1) - 9 \bigl(3 + m^{2}-8 m\bigr)\tilde{\omega }^{4}+ ( m-3) (1 + m) \bigl(4 \tilde{K}_{2} \tilde{\omega } ^{2} \\ &\hphantom{g_{21I}=}{} + 5 d_{2} \mu _{k} \bigl(2 \tilde{K}_{2}^{2} + \tilde{\omega }^{2}\bigr)\bigr) \tilde{\theta }\bigr) - 2 \tilde{K}_{1} \bigl(18 d_{1}\bigl(1+ m^{2}-6 m\bigr) \mu _{k} \tilde{\omega }^{2} \bigl(d_{1}^{2} \mu _{k}^{2} + \tilde{ \omega }^{2}\bigr) \\ &\hphantom{g_{21I}=}{} - 9\tilde{K}_{2} \bigl(3 + m^{2}-8 m\bigr) \tilde{\omega }^{2} \bigl(3 d_{1}^{2} \mu _{k}^{2} + \tilde{\omega }^{2}\bigr) + \tilde{K}_{2} \mu _{k} ( m-3) (1 + m) \bigl(15 d_{1}^{2} d_{2} \mu _{k}^{2}\\ &\hphantom{g_{21I}=}{} + (4 d_{1} + 5 d_{2}) \tilde{\omega } ^{2}\bigr) \tilde{\theta }+ 2 \tilde{K}_{2}^{3} \bigl(9 ( m -1) \tilde{\omega }^{2}+ 5 d_{2}\bigl(m^{2}-2m-3 \bigr)\mu _{k} \tilde{\theta }\bigr)\\ &\hphantom{g_{21I}=}{} + \tilde{K}_{2}^{2} \bigl(9 d_{1} \bigl(9 + m^{2}-14 m\bigr) \mu _{k} \tilde{\omega } ^{2} + 2 (m-3) (1 + m)\tilde{\theta }\bigl(2 \tilde{ \omega }^{2}-15 d_{1} d _{2} \mu _{k}^{2} \bigr)\bigr)\bigr)) \\ &\hphantom{g_{21I}=}{} + \beta (1 + m)^{3} (\tilde{K}_{2}^{2} (1 + m)^{3} \bigl(9(\tilde{\theta }+ d_{1} \mu _{k})^{2} \tilde{\omega }^{2} + 3 \tilde{\omega }^{4} + 10 \bigl(-( \tilde{\theta }+ d_{1} \mu _{k}) \bigl( \tilde{K}_{1}^{2} \\ &\hphantom{g_{21I}=}{}+ \tilde{K}_{1} (\tilde{K}_{2} - 2 d_{1} \mu _{k})+ \mu _{k} \bigl( (d_{2}-d_{1}) \tilde{K}_{2} + \mu _{k}d_{1}^{2}\bigr) \bigr) + ( \tilde{K}_{1} - d_{1} \mu _{k}) \tilde{ \omega }^{2}\bigr) \tilde{\theta } \\ &\hphantom{g_{21I}=}{}+ 10 \tilde{K}_{2} ( \tilde{K}_{1} - d_{1} \mu _{k}) \tilde{\theta }^{2}\bigr) + 2 \tilde{\alpha }( m-3) m (5 \tilde{K}_{1}^{5} + 5 \tilde{K}_{1}^{4}(4 \tilde{K}_{2} - 5 d_{1} \mu _{k})- 5 \mu _{k} (d_{1} \mu _{k}\\ &\hphantom{g_{21I}=}{}- \tilde{K}_{2} )^{3} \bigl(-d_{1} \tilde{K}_{2} + d_{2} \tilde{K}_{2} + d _{1}^{2} \mu _{k} \bigr) + (\tilde{K}_{2} - d_{1} \mu _{k}) \bigl( \tilde{K}_{2} ^{2} + (5 d_{2}-11 d_{1} ) \tilde{K}_{2} \mu _{k} \\ &\hphantom{g_{21I}=}{}+10 d_{1}^{2} \mu _{k}^{2}\bigr) \tilde{\omega }^{2} + (\tilde{K}_{2} - 5 d_{1} \mu _{k}) \tilde{\omega }^{4} - \tilde{K}_{2} \bigl((2 \tilde{K}_{2} +5 d_{1} \mu _{k}) \tilde{ \omega }^{2}+5 d_{1} \mu _{k} ( \tilde{K}_{2} - d _{1} \mu _{k})^{2} \bigr) \tilde{\theta } \\ &\hphantom{g_{21I}=}{}+ 5 \tilde{K}_{1}^{3} \bigl(6 \tilde{K} _{2}^{2}+ 2 \bigl(\tilde{\omega }^{2}+5 d_{1}^{2} \mu _{k}^{2} \bigr) + \tilde{K}_{2} \bigl( (d_{2} -16 d_{1} )\mu _{k} + \tilde{\theta }\bigr)\bigr) + \tilde{K}_{1}^{2} \bigl(20\tilde{K}_{2}^{3} \\ &\hphantom{g_{21I}=}{} - 10 d_{1} \mu _{k} \bigl(5 d_{1}^{2} \mu _{k}^{2} + 3 \tilde{\omega }^{2}\bigr) + 3 \tilde{K}_{2} \bigl(7 \tilde{\omega }^{2} + 5 d_{1} \mu _{k} (8 d_{1} \mu _{k} - d_{2} \mu _{k} - \tilde{\theta })\bigr)\\ &\hphantom{g_{21I}=}{} + 5 \tilde{K}_{2}^{2} \bigl(3 ( d _{2}-6d_{1}) \mu _{k} + 2 \tilde{\theta }\bigr)\bigr) + \tilde{K}_{1} \bigl(5 \tilde{K}_{2}^{4} + 5 \bigl(5 d_{1}^{4} \mu _{k}^{4} + 6 d_{1}^{2} \mu _{k} ^{2} \tilde{\omega }^{2} + \tilde{\omega }^{4}\bigr) \\ &\hphantom{g_{21I}=}{}+ 2 \tilde{K}_{2}^{2} \bigl(6 \tilde{ \omega }^{2}- 5 d_{1}\mu _{k}\bigl(2 \tilde{\theta }+ 3(3 d_{1}-d_{2}) \mu _{k} \bigr)\bigr) + 5 \tilde{K} _{2}^{3} \bigl( ( 3 d_{2} -8 d_{1}) \mu _{k}+ \tilde{\theta }\bigr) \\ &\hphantom{g_{21I}=}{}+ \tilde{K} _{2} \bigl( 15 d_{1}^{2} \mu _{k}^{2} (d_{2} \mu _{k} + \tilde{ \theta })-80 d _{1}^{3} \mu _{k}^{3} - 42 d_{1} \mu _{k} \tilde{\omega }^{2} +5 \tilde{\omega }^{2} (d_{2} \mu _{k}+ \tilde{ \theta })\bigr)\bigr)))). \end{aligned}$$

Let

$$ \begin{aligned} &c_{1}(0)= \frac{i}{2\tilde{\omega }} \biggl(g_{20}g_{11}-2 \vert g_{11} \vert ^{2}- \frac{ \vert g_{02} \vert ^{2}}{3} \biggr)+ \frac{g_{21}}{2}, \qquad \mu _{2}^{*}=- \frac{\operatorname{Re}(c_{1}(0))}{\operatorname{Re}(\lambda '( \tilde{\alpha }))}, \\ &T_{2}=- \frac{1}{\tilde{\omega }} \bigl(\operatorname{Im} \bigl(c_{1}(0)\bigr)+\mu _{2}\bigl(\operatorname{Im}\bigl( \lambda '(\tilde{\alpha })\bigr)\bigr) \bigr), \qquad \beta ^{*}_{2}=2\operatorname{Re}\bigl(c_{1}(0)\bigr). \end{aligned} $$

(3.9)

Then the coefficients in (3.9) determine the properties of the Hopf bifurcation as follows:

• (i) $\mu _{2}^{*}$ determines the direction of the Hopf bifurcation: if $\mu _{2}^{*}> (<) 0$, then the direction of the Hopf bifurcation is forward (backward), that is, the bifurcating periodic solutions exist with $\alpha > (<) \tilde{\alpha }$;

• (ii) $\beta ^{*}_{2}$ determines the stability of the bifurcating periodic solutions on the center manifold: if $\beta ^{*} _{2}< (>) 0$, then the bifurcating periodic solutions are orbital asymptotically stable (unstable);

• (iii) $T_{2}$ determines the period of the bifurcating periodic solutions: if $T_{2}> (<) 0$, then the period increases (decreases).

In particular, when $k=0$, it follows from (3.9) that

$$ \begin{aligned}[b] c_{1}(0)&= \frac{12m}{(1 + m)^{2} (\beta (1+m)-(1-m) )} +i \frac{\beta }{6 (1 + \beta ) ( m-1) (1 + m)^{2} \tilde{\omega }^{3}} \bigl( 2 (1 - 3 m)^{2} \\ &\quad{}+ \beta ^{2} \bigl(7 + m (6 + m) (3+8 m)\bigr) + \beta \bigl( m \bigl(-26 + m (53 + 8 m)\bigr)-7\bigr)\bigr), \end{aligned} $$

(3.10)

and

$$\begin{aligned} &\mu _{2}^{*}= \frac{12m}{ (1-m)- \beta (1+m)}>0, \\ &\beta ^{*}_{2}= \frac{24m}{(1 + m)^{2} ( \beta (1+m)-(1-m) )} < 0, \\ &T_{2}= \frac{2(\beta ^{2} ( m (-36 + m (-51 + 10 m))-7)+ \beta (7 + m (8 + m (-53 + 10 m))) +2 (1 - 3 m)^{2} )}{3 \beta (1 + \beta ) (m-1) (1 + m)^{2} ( \beta (1+m) -1 + m )^{2}}. \end{aligned}$$

This means that when $\alpha =\alpha _{0}$, system (1.2) undergoes a supercritical Hopf bifurcation and the bifurcating periodic solution is spatial homogeneous and orbital asymptotically stable.

Remark 3.1

Compared with [33], here the reason for using α as a bifurcating parameter is that we can give a more detailed analysis of Hopf bifurcation when the spatial homogeneous bifurcating periodic solution comes out, including the accurate bifurcation direction, the certain stability of bifurcating periodic solution, and the computational formula of the tendency of the period.

In order to illustrate our results, we do some numerical simulations for different α. Let

$$ \begin{gathered} d_{1}=3, \qquad d_{2}=0.5,\qquad l=2,\qquad \beta =0.1, \qquad m=0.4< 1,\\ u_{0}=0.2+0.1 \cos x,\qquad v_{0}=0.2+0.1\sin x. \end{gathered} $$

A straightforward calculation leads to the first bifurcating value $\alpha _{0}=1.078$. When $\alpha =0.578<\alpha _{0}$, Fig. 2 shows that the solution of system (1.2) converges to $E^{*}=(0.587,0.587)$. When $\alpha =\alpha _{0}$, Fig. 3 shows that system (1.2) undergoes a Hopf bifurcation and the periodic solution bifurcates from $E^{*}=(0.23, 0.23)$. In this case, using the formula of (3.10), $c_{1}(0)=-5.324-1.965i$ and $T_{2}=3.244>0$ imply that the period of the bifurcating periodic solution increases.

Figure 2

$E^{*}=(0.587,0.587)$ is locally asymptotically stable with $\alpha =0.578$

Figure 3

System (1.2) undergoes a Hopf bifurcation near $E^{*}=(0.23, 0.23)$ with $\alpha =1.078$

4 Nonconstant positive solutions

In this section, we establish the nonexistence and existence of nonconstant positive solution $(u(x), v(x))\in [ C^{2}(\varOmega ) \cap C^{1}(\bar{\varOmega }) ]^{2}$ of system (1.2), where $(u(x), v(x))$ satisfies

$$ \textstyle\begin{cases} d_{1}\triangle {u} +u(1-u)- \frac{\alpha u^{2} v}{u^{2}+m v^{2}}=0, &x\in \varOmega , \\ d_{2}\triangle {v}+\beta v (1- \frac{v}{u} )=0, & x\in \varOmega , \\ \frac{\partial {u}}{\partial {\nu }}= \frac{\partial {v}}{\partial \nu }=0, & x\in \partial {\varOmega }. \end{cases} $$

(4.1)

4.1 A priori estimates of nonnegative solutions

In this subsection, we derive a priori estimates of nonnegative solutions of system (4.1). We first introduce some known results.

Lemma 4.1

(Maximum principle [21, 22])

Assume that $f\in C( \varOmega )$ and $c_{j}\in C(\varOmega )$ with $j=1,2,\ldots , n$.

• (i) If $\omega \in C^{1}(\bar{\varOmega })\cap C^{2}( \varOmega )$ satisfies

  $$ \textstyle\begin{cases} \triangle {\omega }+\sum^{n}_{j=1}c_{j}(x)\omega _{x_{j}}+f(x)\geq 0, &x\in \varOmega , \\ \partial _{\nu }{\omega }\leq 0, & x\in \partial {\varOmega } \end{cases} $$

  and $\omega (x_{0})=\max_{x\in \bar{\varOmega }} \omega (x)$, then $f(x_{0})\geq 0$.

• (ii) If $\omega \in C^{1}(\bar{\varOmega })\cap C^{2}( \varOmega )$ satisfies

  $$ \textstyle\begin{cases} \triangle {\omega }+\sum^{n}_{j=1}c_{j}(x)\omega _{x_{j}}+f(x)\leq 0, &x\in \varOmega , \\ \partial _{\nu }{\omega }\geq 0, & x\in \partial {\varOmega } \end{cases} $$

  and $\omega (x_{0})=\min_{x\in \bar{\varOmega }} \omega (x)$, then $f(x_{0})\leq 0$.

Lemma 4.2

(Harnack inequality [20, 22])

If $u\in C^{2}(\varOmega )\cap C^{1}(\bar{\varOmega })$ is a positive solution of

$$ \textstyle\begin{cases} \triangle u(x)+b(x)u(x)=0, & x\in \varOmega , \\ \partial _{\nu }{u}=0, & x\in \partial {\varOmega }, \end{cases} $$

where $b\in C(\varOmega )\cap L^{\infty }(\varOmega )$, then there exists a positive constant L which depends only on M, satisfying $\|b\|_{\infty }\leq M$, such that

$$ \max_{x\in \bar{\varOmega }}u(x)\leq L \min_{x\in \bar{\varOmega }}u(x). $$

We now establish a priori estimates of nonnegative solutions of system (4.1).

Theorem 4.1

If $(u(x), v(x))$ is a positive solution of system (4.1), then

$$ 0< \min_{x\in \bar{\varOmega }} u(x)\leq \min_{x\in \bar{\varOmega }} v(x) \leq \max_{x\in \bar{\varOmega }} v(x) \leq \max_{x\in \bar{\varOmega }} u(x) \leq 1. $$

Proof

Assume that $(u(x), v(x))$ is a positive solution of system (4.1). From the first equation of system (4.1), we have

$$ d_{1}\triangle {u} +u(1-u)= \frac{\alpha u^{2} v}{u^{2}+m v^{2}}\geq 0. $$

Let $u(x_{0})=\max_{x\in \bar{\varOmega }}u(x)$. From Lemma 4.1, we have $u(x_{0})(1-u(x_{0}))\geq 0$, that is, $\max_{x\in \bar{\varOmega }}u(x) \leq 1$.

Let $v(x_{1})=\max_{x\in \bar{\varOmega }}v(x)$. Then we have

$$ 0\leq \beta v(x_{1}) \biggl(1- \frac{v(x_{1})}{u(x_{1})} \biggr) \leq \beta v(x_{1}) \biggl(1- \frac{v(x_{1})}{u(x_{0})} \biggr), $$

that is, $\max_{x\in \bar{\varOmega }} v(x)\leq \max_{x\in \bar{\varOmega }} u(x)$.

Let $v(x_{2})=\min_{x\in \bar{\varOmega }}v(x)$. It follows from Lemma 4.1 that

$$ \beta v(x_{2}) \biggl(1- \frac{v(x_{2})}{u(x_{2})} \biggr)\leq 0, $$

that is, $\min_{x\in \bar{\varOmega }} v(x)\geq u(x_{2})\geq \min_{x\in \bar{\varOmega }} u(x)$. □

Theorem 4.2

Assume that d̄ is a positive constant. If $d_{1}>\bar{d}$, then there is a positive constant L such that each positive solution $(u(x),v(x))$ of system (4.1) satisfies

$$ 0< \max_{x\in \bar{\varOmega }} u(x)\leq L\min_{x\in \bar{\varOmega }}u(x). $$

Proof

Let

$$ b(x)= \frac{u(x)}{d_{1}} \biggl(1-u(x)- \frac{\alpha u(x) v(x)}{u^{2}(x)+mv^{2}(x)} \biggr). $$

Then the first equation of system (4.1) can be written as

$$ \triangle {u(x)} +b(x)u(x)=0. $$

(4.2)

From Theorem 4.1, we have $b(x)\in C(\varOmega )\cap L^{\infty }(\varOmega )$. It follows from Lemma 4.2 that the conclusion of Theorem 4.2 holds. □

From Theorems 4.1 and 4.2, we conclude that there exists a positive constant $\underline{\kappa }$ such that

$$ \min_{x\in \bar{\varOmega }} u(x)\geq \underline{\kappa }, \qquad \min_{x\in \bar{\varOmega }} v(x)\geq \underline{\kappa } $$

(4.3)

for any $d_{1}\geq \bar{d}$ and any positive solution $(u(x), v(x))$ of system (4.1). By using the standard Schauder theory for elliptic equations, we also conclude that there exists a positive constant κ̄ such that

$$ \Vert u \Vert _{2+\gamma }\leq \bar{\kappa }, \qquad \Vert v \Vert _{2+\gamma }\leq \bar{\kappa } $$

(4.4)

for all $d_{1},d_{2}\geq \hat{d}$ and each positive solution $(u,v)\in C^{2+\gamma }(\bar{\varOmega })\times C^{2+\gamma }(\bar{ \varOmega })$ of system (4.1), where d̂ is a positive constant and $\gamma \in (0,1)$.

4.2 Nonexistence of nonconstant positive solutions

In this subsection, we explore the nonexistence of nonconstant positive solutions of system (4.1) for different diffusion coefficients $d_{1}$, $d_{2}$.

We first show that if diffusion coefficients of predator and prey are both sufficiently large, then system (4.1) has no nonconstant positive solutions. Let $\bar{u}=| \varOmega |^{-1}\int _{\varOmega }u(x) \,\mathrm{d}x$ and $\bar{v}=| \varOmega |^{-1}\int _{\varOmega }v(x)\,\mathrm{d}x$ for any positive solution $(u,v)$ of system (4.1). It is clear that $\int _{\varOmega }(u- \bar{u})\,\mathrm{d}x=\int _{\varOmega }(v-\bar{v})\,\mathrm{d}x=0$. Multiplying the first equation of system (4.1) by $u-\bar{u}$ and the second equation of system (4.1) by $v-\bar{v}$, then integrating on Ω, respectively, we have

$$\begin{aligned}& \begin{aligned}[b] d_{1} \int _{\varOmega } \bigl\vert \nabla (u-\bar{u}) \bigr\vert ^{2}\,\mathrm{d}x&= \int _{ \varOmega }(u-\bar{u})u(1-u)\,\mathrm{d}x- \int _{\varOmega }(u-\bar{u}) \frac{\alpha u^{2}v}{u^{2}+m v^{2}}\,\mathrm{d}x \\ &= L_{1}(u,\bar{u})+L_{2}(u,\bar{u},v), \end{aligned} \end{aligned}$$

(4.5)

$$\begin{aligned}& \begin{aligned}[b] d_{2} \int _{\varOmega } \bigl\vert \nabla (v-\bar{v}) \bigr\vert ^{2}\,\mathrm{d}x&=\beta \int _{\varOmega }(v-\bar{v}) \biggl[v \biggl(1- \frac{v}{u} \biggr)-\bar{v} \biggl(1- \frac{\bar{v}}{\bar{u}} \biggr) \biggr]\,\mathrm{d}x \\ &=\beta \int _{\varOmega }(v-\bar{v})^{2}\,\mathrm{d}x+L_{3}(u,v, \bar{v}), \end{aligned} \end{aligned}$$

(4.6)

and the following.

Theorem 4.3

Let $d_{0}$ be a positive constant. Then system (4.1) has no nonconstant positive solutions for any $d_{1}, d_{2}>d_{0}$.

Proof

Assume that $(u,v)$ is a positive solution of system (4.1). From Theorem 4.1 and (4.3), we have $\underline{\kappa }\leq u$, $v\leq 1$ and $\underline{\kappa }\leq \bar{u}$, $\bar{v}\leq 1$ for any $x\in \bar{\varOmega }$. Then

$$\begin{aligned}& \begin{aligned} L_{1}(u,\bar{u})&= \int _{\varOmega }(u-\bar{u})\bigl[u(1-u)-\bar{u}(1-\bar{u})\bigr] \, \mathrm{d}x \\ &= \int _{\varOmega } (u-\bar{u})^{2}(1-u-\bar{u})\,\mathrm{d}x \\ &\leq \int _{\varOmega } (u-\bar{u})^{2}\,\mathrm{d}x, \end{aligned} \\& \begin{aligned} L_{2}(u,\bar{u},v)&=-\alpha \int _{\varOmega }(u-\bar{u}) \biggl( \frac{ u^{2}v}{u^{2}+m v^{2}}- \frac{\bar{u}^{2}\bar{v}}{\bar{u}^{2}+m \bar{v}^{2}} \biggr)\,\mathrm{d}x \\ &=-\alpha \int _{\varOmega } \frac{(u-\bar{u})^{2} mv\bar{v}^{2}(u+\bar{u})}{(u^{2}+m v^{2}) ( \bar{u}^{2}+m \bar{v}^{2})}\,\mathrm{d}x\\&\quad{} -\alpha \int _{\varOmega } \frac{(u-\bar{u})(v-\bar{v})\bar{u}^{2}}{(u^{2}+m v^{2}) (\bar{u}^{2}+m \bar{v}^{2})}\bigl(u^{2}-mv\bar{v} \bigr)\,\mathrm{d}x \\ &\leq \alpha \int _{\varOmega } \frac{(u-\bar{u})(v-\bar{v})\bar{u}^{2}}{(u^{2}+m v^{2}) (\bar{u}^{2}+m \bar{v}^{2})}\bigl(mv\bar{v}- u^{2} \bigr)\,\mathrm{d}x \\ &\leq \frac{\alpha }{(1+m)\underline{\kappa }^{4}} \int _{\varOmega }(u-\bar{u}) (v- \bar{v}) \,\mathrm{d}x \\ &\leq B_{1} \biggl( \int _{\varOmega }(u-\bar{u})^{2}\,\mathrm{d}x+ \int _{\varOmega }(v-\bar{v})^{2}\,\mathrm{d}x \biggr), \end{aligned} \\& \begin{aligned} L_{3}(u,v,\bar{v})&=-\beta \int _{\varOmega }(v-\bar{v}) \biggl( \frac{v^{2}}{u^{2}}- \frac{\bar{v}^{2}}{\bar{u}^{2}} \biggr)\,\mathrm{d}x \\ &=-\beta \int _{\varOmega }(v-\bar{v})^{2} \frac{v+\bar{v}}{u}\, \mathrm{d}x+\beta \int _{\varOmega } \frac{\bar{v}^{2}}{u\bar{u}}(u-\bar{u}) (v-\bar{v})\, \mathrm{d}x \\ &\leq B_{2} \biggl( \int _{\varOmega }(u-\bar{u})^{2}\,\mathrm{d}x+ \int _{\varOmega }(v-\bar{v})^{2}\,\mathrm{d}x \biggr), \end{aligned} \end{aligned}$$

where $B_{1}=\alpha /(2(1+m)\underline{\kappa }^{4})$ and $B_{2}= \beta /(2\underline{\kappa }^{2})$. It follows from (4.5) and (4.6) that

$$ d_{1} \int _{\varOmega } \bigl\vert \nabla (u-\bar{u}) \bigr\vert ^{2}\,\mathrm{d}x\leq (1+B_{1}) \int _{\varOmega }(u-\bar{u})^{2} \,\mathrm{d}x +B_{1} \int _{\varOmega }(v- \bar{v})^{2}\,\mathrm{d}x $$

(4.7)

and

$$ d_{2} \int _{\varOmega } \bigl\vert \nabla (v-\bar{v}) \bigr\vert ^{2}\,\mathrm{d}x\leq (\beta +B _{2}) \int _{\varOmega }(v-\bar{v})^{2}\,\mathrm{d}x+B_{2} \int _{\varOmega }(u- \bar{u})^{2}\,\mathrm{d}x. $$

(4.8)

Combining (4.7) and (4.8) and applying the Poincaré inequality, we have

$$\begin{aligned} &d_{1} \int _{\varOmega } \bigl\vert \nabla (u-\bar{u}) \bigr\vert ^{2}\,\mathrm{d}x+d_{2} \int _{\varOmega } \bigl\vert \nabla (v-\bar{v}) \bigr\vert ^{2}\,\mathrm{d}x \\ &\quad \leq (1+B_{1}+B_{2}) \int _{\varOmega }(u-\bar{u})^{2}\,\mathrm{d}x+( \beta +B_{1}+B_{2}) \int _{\varOmega }(v-\bar{v})^{2}\,\mathrm{d}x \\ &\quad \leq \frac{B}{\mu _{1}} \biggl( \int _{\varOmega } \bigl\vert \nabla (u-\bar{u}) \bigr\vert ^{2}\,\mathrm{d}x+ \int _{\varOmega } \bigl\vert \nabla (v-\bar{v}) \bigr\vert ^{2}\,\mathrm{d}x \biggr), \end{aligned}$$

where $B=\max \{1+B_{1}+B_{2}, \beta +B_{1}+B_{2} \}$. If $\min \{d_{1}, d_{2} \}>d_{0}:=B/\mu _{1}$, then $\nabla (u-\bar{u})=\nabla (v- \bar{v})=0$ for any $x\in \varOmega$, which implies that u and v are both constants when $d_{1}, d_{2}>d _{0}$. This completes the proof. □

We now prove that if the ability of the predators to hunt is strong and the predators move slow or the preys move fast, then system (4.1) has no nonconstant positive solutions.

Lemma 4.3

If $(u,v)$ is a nonconstant positive solution of system (4.1), then

$$ 0< \bar{\eta }:=\min_{x\in \bar{\varOmega }} \frac{v(x)}{u(x)}< 1. $$

Proof

If the conclusion is not true, then $v(x)\geq \not \equiv u(x)$ for any $x\in \bar{\varOmega }$. Integrating the second equation of system (4.1) on Ω, we have

$$ 0=\beta \int _{\varOmega } v(x) \biggl(1- \frac{v(x)}{u(x)} \biggr)\, \mathrm{d}x< 0, $$

which is a contradiction. □

Theorem 4.4

If $\alpha \geq 1+m$ and $d_{2}/d_{1}\leq \beta$, then system (4.1) has no nonconstant positive solutions.

Proof

Denote $y_{\eta }(x)=v(x)-\eta u(x)$ for any $\eta \in (0,1]$ and

$$ \delta (u,y_{\eta },\eta )= \frac{1}{d_{2}}\beta (y_{\eta }+ \eta u) \biggl(1- \frac{y_{\eta }+\eta u}{u} \biggr)- \frac{\eta }{d_{1}} \biggl(u(1-u)- \frac{\alpha u^{2}(y_{\eta }+\eta u)}{u^{2}+m(y_{\eta }+\eta u)^{2}} \biggr). $$

Then

$$ \delta (u,0,\eta )=\eta u \biggl[ \frac{\beta }{d_{2}}(1-\eta )- \frac{1}{d_{1}} \biggl(1-u- \frac{\alpha \eta }{1+m\eta ^{2}} \biggr) \biggr]=\eta u \rho (u, \eta ), $$

where

$$ \rho (u,\eta )= \frac{\beta }{d_{2}}(1-\eta )- \frac{1}{d_{1}} \biggl(1-u- \frac{\alpha \eta }{1+m\eta ^{2}} \biggr). $$

Note that if $d_{2}/d_{1}\leq \beta$ and $\alpha \geq 1+m$, then

$$ \rho (u,0)= \frac{u}{d_{1}}+ \frac{\beta }{d_{2}}- \frac{1}{d_{1}}>0, \qquad \rho (u,1)= \frac{u}{d_{1}}+ \frac{1}{d_{1}} \biggl( \frac{\alpha }{1+m}-1 \biggr)>0 $$

and

$$\begin{aligned} &\rho _{\eta }'(u,\eta )= \frac{u}{d_{1}}+ \frac{1}{d_{1}d_{2}(1+m\eta ^{2})^{2}} \bigl(-d_{1}\beta m^{2} \eta ^{4}+(2d _{1} \beta -d_{2} \alpha )m \eta ^{2}+d_{2} \alpha -d_{1} \beta \bigr)>0, \\ &\rho _{u}'(u,\eta )= \frac{1}{d_{1}}>0 \end{aligned}$$

for any $u>0$ and $\eta \in (0,1]$, which means $\delta (u, 0,\eta )>0$ for any $u>0$ and $\eta \in (0,1]$.

It follows from Lemma 4.3 that there exists $\bar{x} \in \bar{\varOmega }$ such that $0<\bar{\eta }=u(\bar{x})/v(\bar{x})<1$. Let $y_{\bar{\eta }}(x)=v(x)-\bar{\eta } u(x)$. Then

$$ \textstyle\begin{cases} -\triangle {y_{\bar{\eta }}}=\delta (u,y_{\bar{\eta }},\bar{\eta })>0, &x\in \varOmega , \\ {\partial _{\nu }{y_{\bar{\eta }}}}\geq 0, & x\in \partial {\varOmega }. \end{cases} $$

(4.9)

From the strong maximum principle and Hopf lemma, we get $y_{\bar{ \eta }}(x)>0$ on Ω̄, which is contrary to $y_{\bar{ \eta }}(\bar{x})=0$. Therefore, system (4.1) has no nonconstant positive solutions. □

We next show that if the ability of the predators to hunt is weak and the preys move fast, then system (4.1) has no nonconstant positive solution. For $0<\gamma <1$, we let

$$ \begin{gathered} {\mathbb{Y}}_{1}= \biggl\{ u\in C^{\gamma }(\bar{\varOmega }): \int _{\varOmega }u\,\mathrm{d}x=0 \biggr\} ,\\ {\mathbb{Y}}_{2}= \bigl\{ u\in C^{2+\gamma }(\bar{\varOmega }): \partial _{\nu } u=0 \text{ on } \partial \varOmega \bigr\} ,\qquad {\mathbb{Y}}_{3}={\mathbb{Y}}_{1} \cap {\mathbb{Y}}_{2}, \end{gathered} $$

$u=\tau +\omega$ with $\tau \in {\mathbb{R}}$ and $\omega \in {\mathbb{Y}} _{3}$. Let $\xi =d_{1}^{-1}$ and

$$\begin{aligned} h_{1}(\xi , \tau , \omega , v)&= \frac{1}{ \vert \varOmega \vert } \int _{\varOmega } \biggl[(\tau +\omega ) (1-\tau - \omega )- \frac{\alpha (\tau +\omega )^{2} v}{(\tau +\omega )^{2}+mv^{2}} \biggr] \,\mathrm{d}x, \\ h_{2}(\xi , \tau , \omega , v)&=\triangle \omega +\xi (\tau +\omega ) (1-\tau -\omega )- \frac{\xi \alpha (\tau +\omega )^{2} v}{(\tau +\omega )^{2}+mv^{2}}- \xi f_{1}(\xi , \tau , \omega , v), \\ h_{3}(\xi , \tau , \omega , v)&=d_{2}\triangle v+\beta v \biggl(1- \frac{v}{\tau +\omega } \biggr), \\ H(\xi , \tau , \omega , v)&=(h_{1},h_{2},h_{3})^{T}( \xi , \tau , \omega , v). \end{aligned}$$

It is clear that $H:{\mathbb{R}}^{2}\times {\mathbb{Y}}_{3}\times {\mathbb{Y}}_{2}\rightarrow {\mathbb{R}}\times {\mathbb{Y}}_{1}\times C^{\gamma }(\bar{\varOmega })$ and $(u,v)$ is a solution of system (4.1) if and only if $H(\xi , \tau , \omega , v)=0$. It follows that

$$ \varPhi (\tau ,\omega ,v):=H_{(\tau ,\omega ,v)}(0, \theta , 0, \theta )= \begin{pmatrix} { \vert \varOmega \vert ^{-1}} \int _{\varOmega }(K_{1}\tau +K_{1}\omega +K_{2}v )\,\mathrm{d}x \\ \triangle \omega \\ d_{2}\triangle v-\beta v+\beta \tau +\beta \omega \end{pmatrix} , $$

where $\theta =1-\alpha /(1+m)$, $K_{1}$ and $K_{2}$ are defined in (2.6). A straightforward calculation yields the following.

Lemma 4.4

Φ is an isomorphism.

Proof

To obtain the conclusion, we only need to prove that

$$\begin{aligned}& { \vert \varOmega \vert ^{-1}} \int _{\varOmega }(K_{1} \tau +K_{1}\omega +K_{2}v)\,\mathrm{d}x=\bar{ \tau }, \end{aligned}$$

(4.10)

$$\begin{aligned}& \triangle \omega =\bar{\omega }, \qquad x\in \varOmega , \qquad { \partial _{\nu } \omega }|_{\partial \varOmega }=0, \qquad \int _{\varOmega }\omega \,\mathrm{d}x=0, \end{aligned}$$

(4.11)

$$\begin{aligned}& \textstyle\begin{cases} d_{2}\triangle v-\beta v=\bar{v}-\beta (\tau +\omega ),&x\in \varOmega , \\ {\partial _{\nu }{v}}=0,&x\in \partial {\varOmega } \end{cases}\displaystyle \end{aligned}$$

(4.12)

has a unique solution for any given $(\bar{\tau }, \bar{\omega }, \bar{v})\in {\mathbb{R}}\times {\mathbb{Y}}_{1}\times C^{\gamma }(\bar{ \varOmega })$.

It is easy to see that system (4.11) has a unique solution $\omega \in {\mathbb{Y}}_{3}$ since $\bar{\omega }\in {\mathbb{Y}} _{1}$. From (4.10) and (4.11), we have

$$ \int _{\varOmega }v\,\mathrm{d}x={(\bar{\tau }-\tau K_{1})} {K_{2}}^{-1} \vert \varOmega \vert . $$

Integrating the first equation of system (4.12) over Ω, we obtain

$$ \tau ={ \biggl( \int _{\varOmega }\bar{v}\,\mathrm{d}x+\bar{\tau }\beta \vert \varOmega \vert \biggr)} {\bigl( \vert \varOmega \vert \beta (K_{1}+K_{2}) \bigr)^{-1}}. $$

(4.13)

Then system (4.12) has a unique solution v, and v satisfies

$$ \beta \int _{\varOmega }v\,\mathrm{d}x=\beta \tau \vert \varOmega \vert - \int _{\varOmega }\bar{v}\,\mathrm{d}x. $$

(4.14)

This means that Φ is an isomorphism. □

Let $d_{1}^{i}\in (0, +\infty )$ and $(u^{i}, v^{i})$ be positive solutions of system (4.1) with $d_{1}=d_{1}^{i}$. By using a method similar to that mentioned in [23], we have the following lemma.

Lemma 4.5

Assume that $\alpha <1+m$ and $(u^{i}, v^{i})\rightarrow (\tilde{u}, \tilde{v})$ uniformly on Ω̄ as $d_{1}^{i}\rightarrow \tilde{d}_{1}\in [0, +\infty ]$. If ũ and ṽ are positive constants, then $(\tilde{u}, \tilde{v})=(\theta ,\theta )$.

Theorem 4.5

Assume that $\alpha <1+m$ and $\tilde{d}_{1}\gg 1$ is a fixed constant. Then system (4.1) has no nonconstant positive solution for any $d_{1}>\tilde{d}_{1}$.

Proof

We claim that system (4.1) has only a positive solution $(\theta ,\theta )$ in a small neighborhood of $(\theta ,\theta )$. In fact, it follows from Lemma 4.4 that $\varPhi ^{-1}$ exists and is a bounded linear operator. By using the implicit function theorem, we conclude that there exists a constant $\varepsilon >0$ such that, for all $0<\xi <\varepsilon$, $H(\xi ,\tau ,\omega ,v)=0$ has a unique positive solution $(\theta ,0,\theta )$ in the small neighborhood $B_{\varepsilon }(\theta ,0,\theta )$. This implies that when $d_{1}>1/\varepsilon$, system (4.1) has only a positive solution $(\theta ,\theta )$ in $B_{\varepsilon }(\theta ,\theta )$.

Assume that $(u^{i}, v^{i})$ is the nonconstant positive solution of system (4.1) with $d_{1}=d_{1}^{i}\in (0, +\infty )$ and $d_{1}^{i}\rightarrow +\infty$. It follows from (4.4) that $(u^{i},v^{i})\rightarrow (\hat{u}, \hat{v})$ in $[C^{2}(\bar{ \varOmega }) ]^{2}$ as $d_{1}^{i}\rightarrow +\infty$. From Theorem 4.1 and (4.3), we conclude that $(\hat{u},\hat{v})$ is bounded and $\hat{u}>0$ satisfies

$$ -\triangle \hat{u}=0, \qquad x\in \varOmega , \qquad {\partial _{\nu } \hat{u}}=0, \qquad x\in \partial \varOmega . $$

This means that û is a positive constant. Substituting it into the second equation of system (4.1), we get

$$ \triangle v+\beta v (1-{v}/{\hat{u}} )=0, \qquad x\in \varOmega , \qquad \partial _{\nu } v=0, \qquad x\in \partial \varOmega , $$

which implies that $v=\hat{u}$. It follows from Lemma 4.5 and $(u^{i},v^{i})\rightarrow (\hat{u},\hat{v})$ that $(\hat{u},\hat{v}) \equiv (\theta ,\theta )$ and there exists $i_{0}$ such that $(u^{i},v^{i})=(\theta ,\theta )$ for any $i\geq i_{0}$ and each $d_{1}^{i}>1/\varepsilon$. This is a contradiction to that $(u^{i},v^{i})$ is a nonconstant positive solution of system (4.1). □

4.3 Existence of nonconstant positive solutions

This subsection is devoted to investigating the existence of nonconstant positive solutions of system (4.1) for $\alpha \in ((1+m)^{2}/2, 1+m)$ by using Leray–Schauder degree theory. There exists a unique positive constant solution $E^{*}=(\theta ,\theta )$ if $\alpha <1+m$. Let

$$W(U):=\left(\begin{array}{c}u(1-u)-\frac{\alpha u^{2}v}{u^2+m{v}^2}\\ \beta v(1-\frac{v}{u})\end{array}\right)$$

(4.15)

with $U=(u,v)^{T}\in {\mathbb{X}}$ and $(I-\triangle )^{-1}$ be the inverse of $I-\triangle$. Then system (4.1) can reduce to

$$ \mathcal{G}(d_{1},d_{2}, U):=U-(I-\triangle )^{-1} \bigl\{ D^{-1}W(U)+U \bigr\} =0, $$

(4.16)

where $I-\triangle$ satisfies the homogeneous Neumann boundary condition. Frechét derivative of system (4.16) with respect to U at $(\theta ,\theta )$ is

$$ \mathcal{G}_{U}(d_{1},d_{2},\theta ,\theta )=I-(I-\triangle )^{-1} \bigl\{ D^{-1}W_{U} (\theta , \theta )+I \bigr\} =0. $$

Obviously, ζ is an eigenvalue of $\mathcal{G}_{U}(d_{1},d_{2}, \theta ,\theta )$ on $X_{i}$ with $i\in {\mathbb{N}}_{0}$ if and only if $\zeta (1+\mu _{i})$ is an eigenvalue of the matrix

$$L_i:={\mu }_{i}I-D^{-1}{W}_U(\theta ,\theta )=\left(\begin{array}{cc}{\mu }_i-K_1/d_1& -K_2/d_1\\ -\beta /d_2& {\mu }_i+\beta /d_2\end{array}\right),$$

(4.17)

where $K_{1}$ and $K_{2}$ are defined in (2.6). Then

$$ \det L_{i}= \frac{1}{d_{1}d_{2}} \bigl(d_{1}d_{2} \mu ^{2}_{i}-(d_{2} K_{1}-\beta d _{1})\mu _{i}+\beta \theta \bigr)= \frac{1}{d_{1}d_{2}}S(d_{1},d_{2}, \mu _{i}), $$

where

$$ S(d_{1},d_{2},\mu ): =d_{1}d_{2} \mu ^{2}-(d_{2} K_{1}-\beta d_{1}) \mu + \beta \theta . $$

(4.18)

In order to obtain the existence of nonconstant positive solutions of system (4.1), we need the following two lemmas.

Lemma 4.6

If $m<1$, $\alpha \in ((1+m)^{2}/2, 1+m)$ and $d_{1}/d_{2}< d_{-}$, then $S(d_{1},d_{2},\mu )=0$ has two positive roots

$$ \mu _{\pm }(d_{1},d_{2})= \frac{d_{2}K_{1}-\beta d_{1}\pm \sqrt{\Delta _{S}}}{2d_{1}d_{2}}, $$

(4.19)

where

$$ \Delta _{S}=(d_{2}K_{1}-\beta d_{1})^{2}-4d_{1}d_{2}\beta \theta , \quad \textit{and} \quad d_{-}= \bigl(2\theta +K_{1}- 2\sqrt{ \theta (\theta +K_{1})} \bigr) /\beta . $$

Proof

It is not difficult to show that $S(d_{1},d_{2},\mu )=0$ has two positive roots if and only if $\Delta _{S}\geq 0$ and

$$ l(d_{1},d_{2})= \frac{K_{1} d_{2}-\beta d_{1}}{2d_{1}d_{2}} = \frac{1}{2d_{1}d_{2}} \biggl(d_{2} \biggl( \frac{2\alpha }{(1+m)^{2}}-1 \biggr)-d_{1}\beta \biggr)>0. $$

A direct calculation gives

$$\begin{aligned} \Delta _{S} &=\beta ^{2}d_{1}^{2}+2 \beta \biggl( \frac{2\alpha m}{(1+m)^{2}}-1 \biggr)d_{1}d_{2}+K_{1}^{2}d_{2}^{2} \\ &=d_{2}^{2} \biggl(\beta ^{2} \biggl( \frac{d_{1}}{d_{2}} \biggr)^{2}+2\beta \biggl( \frac{2\alpha m}{(1+m)^{2}}-1 \biggr) \biggl( \frac{d_{1}}{d_{2}} \biggr)+K_{1}^{2} \biggr). \end{aligned}$$

Note that if $\alpha >(1+m)^{2}/2$ and $d_{1}/d_{2}< K_{1}/\beta$, then $l(d_{1},d_{2})>0$. Combining with $\alpha <1+m$ implies that $m<1$. This means that $2\alpha m/(1+m)^{2}-1<0$. Let

$$ r(d)=\beta ^{2}d^{2}+2\beta \biggl( \frac{2\alpha m}{(1+m)^{2}}-1 \biggr)d+K_{1}^{2}. $$

The discriminant of the roots of $r(d)=0$ is $\Delta _{r}=16\beta ^{2} \theta (\theta +K_{1}) >0$. Hence, $r(d)=0$ has two positive roots

$$ d_{\pm }= \frac{1}{\beta } \bigl(2\theta +K_{1}\pm 2\sqrt{\theta (\theta +K _{1})} \bigr). $$

(4.20)

When $d< d_{-}$ or $d>d_{+}$, we get $r(d)>0$. On the other hand, $r({K_{1}}/{\beta })=-4K_{1}\theta <0$. That is, if $d_{1}/d_{2}< d _{-}$, then $\Delta _{S}>0$ and $l(d_{1},d_{2})>0$. This shows that (4.19) holds. □

For a fixed $d_{1}>0$, if $d_{2}$ is sufficiently large, then $d_{1}/d_{2}< d_{-}$ holds. Let

$$ \mathcal{R}(d_{1},d_{2})= \bigl\{ \mu \geq 0: \mu _{-}(d_{1},d_{2}) < \mu < \mu _{+}(d_{1},d_{2}) \bigr\} . $$

From (4.19), we have

$$ \lim_{d_{2}\rightarrow +\infty }\mu _{-}(d_{1},d_{2})=0, \qquad \lim_{d_{2}\rightarrow +\infty }\mu _{+}(d_{1},d_{2})={K_{1}}/{d_{1}}>0. $$

(4.21)

Lemma 4.7

([23, 29])

If $S(d_{1},d_{2},\mu _{i})\neq 0$ for all $\mu _{i}\in \varLambda$, then $\operatorname{index}(\mathcal{G}(d _{1},d_{2},\cdot ),(\theta ,\theta ))=(-1)^{\sigma }$, where $\sigma =\sum_{\mu _{i}\in \mathcal{R}(d_{1},d_{2}) \cap \varLambda } m(\mu _{i})$ when $\mathcal{R}(d_{1},d_{2})\cap \varLambda \neq \phi$ and $\sigma =0$ when $\mathcal{R}(d_{1},d_{2})\cap \varLambda =\phi$. In particular, if $S(d_{1},d_{2},\mu )>0$ for all $\mu \geq 0$, then $\sigma =0$.

Theorem 4.6

Assume that $d_{1}$ and β are fixed positive constants and $m<1$, $\alpha \in ((1+m)^{2}/2, 1+m)$, $K_{1}/d_{1}\in (\mu _{k},\mu _{k+1})$ for some $k\in {\mathbb{N}}$. If $\sum^{k}_{i=1}m(\mu _{i})$ is odd, then there exists a positive constant $\hat{d}_{2}$ such that, for any $d_{2}\geq \hat{d}_{2}$, (4.1) has at least one nonconstant positive solution.

Proof

From (4.21) and $K_{1}/d_{1}\in (\mu _{k},\mu _{k+1})$, there exists $\bar{d}_{2}\gg 1$ such that, for any $d_{2}>\bar{d}_{2}$,

$$ 0< \mu _{-}(d_{1},d_{2})< \mu _{1}< \cdots < \mu _{k}< \mu _{+}(d_{1},d_{2})< \mu _{k+1}. $$

(4.22)

It follows from Theorem 4.3 that system (4.1) has no nonconstant positive solutions for any $d_{1},d_{2}>d_{0}$. We choose $\hat{d}_{1}>d_{0}$ and $\hat{d}_{2}>\max \{\bar{d}_{2}, d_{0}\}$ such that $K_{1}/\hat{d}_{1}<\mu _{1}$ and

$$ 0< \mu _{-}(\hat{d}_{1}, \hat{d}_{2})< \mu _{+}( \hat{d}_{1}, \hat{d}_{2})< \mu _{1}. $$

(4.23)

If the conclusion of Theorem 4.6 is not true, then there is some $d_{2}$ such that system (4.1) has no nonconstant positive solutions for $d_{2}\geq \hat{d}_{2}$. For $t\in [0,1]$, we let $D_{t}=\operatorname{diag}(td_{1}+(1-t)\hat{d}_{1}, td_{2}+(1-t)\hat{d}_{2})$ and consider the following system:

$$ \textstyle\begin{cases} -D_{t}\triangle U=W(U), &x\in \varOmega , \\ {\partial _{\nu }U}=0, &x\in \partial \varOmega , \end{cases} $$

(4.24)

where $W(U)$ is defined in (4.15). It is clear that (4.24) is equivalent to

$$ \varPsi (U,t)=U-(I-\triangle )^{-1} \bigl\{ D^{-1}_{t} W(U)+U \bigr\} =0, \quad U\in {\mathbb{X}}. $$

Note that $\varPsi (U,1)=\mathcal{G}(d_{1},d_{2},U)$, $\varPsi (U,0)= \mathcal{G}(\hat{d}_{1},\hat{d}_{2},U)$ and

$$\begin{aligned} &\mathcal{G}_{U}(d_{1},d_{2},\theta ,\theta )=I-(I-\triangle )^{-1} \bigl\{ \operatorname{diag}(d_{1},d_{2})^{-1}W_{U}( \theta , \theta )+I \bigr\} =0, \\ &\mathcal{G}_{U}(\hat{d}_{1},\hat{d}_{2},\theta , \theta )=I-(I- \triangle )^{-1} \bigl\{ \operatorname{diag}( \hat{d}_{1},\hat{d}_{2})^{-1}W _{U}( \theta , \theta )+I \bigr\} =0. \end{aligned}$$

It follows from Theorems 4.5 and 4.3 that $\varPsi (U,1)=0$ and $\varPsi (U,0)=0$ have no nonconstant positive solutions.

By using (4.22) and (4.23), we have

$$ \mathcal{R}(d_{1},d_{2})\cap \varLambda =\{\mu _{1}, \mu _{2},\ldots ,\mu _{k}\}, \qquad \mathcal{R}( \hat{d}_{1},\hat{d}_{2})\cap \varLambda =\phi , $$

which implies that

$$ \operatorname{index} \bigl(\varPsi (\cdot ,1),(\theta ,\theta ) \bigr)=(-1)^{ \sum ^{k}_{i=1}m(\mu _{i})}=-1, \qquad \operatorname{index} \bigl(\varPsi (\cdot ,0),( \theta ,\theta ) \bigr)=(-1)^{0}=1. $$

From Theorem 4.1 and (4.3), we have $\underline{ \kappa }/2< u$, $v<2$ for any solution $(u,v)$ of system (4.1) on Ω̄. Let

$$ \varTheta = \bigl\{ (u,v)^{T}\in {\mathbb{X}}: \underline{\kappa }/2< u,v< 2, x \in \bar{\varOmega } \bigr\} . $$

Then $\varPsi (U,t)\neq 0$ on ∂Θ for all $t\in [0,1]$. It follows from the homotopy invariance of Leray–Schauder degree that

$$ \deg \bigl(\varPsi (\cdot ,0), \varTheta ,0\bigr)=\deg \bigl(\varPsi ( \cdot ,1), \varTheta ,0\bigr). $$

(4.25)

Note that $\varPsi (U,0)=0$ and $\varPsi (U,1)=0$ have only the constant solution $(\theta ,\theta )$ in Θ and hence,

$$\begin{aligned} &\deg \bigl(\varPsi (\cdot ,0), \varTheta ,0\bigr)=\operatorname{index} \bigl(\varPsi ( \cdot ,0),(\theta ,\theta ) \bigr)=1, \\ &\deg \bigl(\varPsi (\cdot ,1), \varTheta ,0\bigr)=\operatorname{index} \bigl(\varPsi ( \cdot ,1),(\theta ,\theta ) \bigr)=-1, \end{aligned}$$

which is a contradiction to (4.25). The proof is complete. □

From Theorem 4.6, if $d_{2}/d_{1}$ is large enough and $m<1$, then Fig. 4 shows that system (4.1) has a nonconstant positive solution.

Figure 4

The nonconstant positive solution of system (1.2). Here $d_{1}=0.005$, $d_{2}=20$, $\alpha =1$, $m=0.4$, $\beta =0.1$