1 Introduction

In the ecological environment, schooling behavior among predators widely exists, such as wolves, African wild dogs, and lions [13]. In [4], Cosner et al. proposed the following functional response

$$ \varphi (u,v)=\frac{C e_{0} u v}{1 +hCe_{0} uv}.$$

The biological meanings of the parameters are given in Table 1. Unlike the traditional functional response (Holling I–III [5]), it is dependent on predator density and increases with prey and predator densities. This functional response can reflect the schooling behavior among predators. Incorporating this functional response, Ryu et al. [6] studied the following model:

$$ \begin{aligned} \textstyle\begin{cases} \frac{du}{dt}= r u (1-\frac{u}{K} )- \frac{C e_{0} u v^{2}}{1 +hCe_{0} uv}, \\ \frac{dv}{dt}=\frac{\epsilon Ce_{0} uv^{2}}{1 +hCe_{0} uv}-\mu v. \end{cases}\displaystyle \end{aligned} $$
(1.1)

By the scaling

$$ \begin{aligned} &rt=\bar{t}, \qquad \frac{u}{K}=\bar{u},\qquad hCe_{0}Ky=\bar{v}, \\ & \frac{1}{Ce_{0}(hK)^{2}r}=\alpha , \qquad \frac{\epsilon}{rh}=\beta ,\qquad \frac{\mu h}{\epsilon}=\gamma , \end{aligned}$$
(1.2)

model (1.1) changed to (dropping the bars)

$$ \begin{aligned} \textstyle\begin{cases} \frac{du}{dt}= u (1-u )-\frac{\alpha u v^{2}}{1 +uv}, \\ \frac{dv}{dt}=\beta (\frac{ uv^{2}}{1 +uv}-\gamma v ). \end{cases}\displaystyle \end{aligned} $$
(1.3)

All parameters are positive. They mainly studied the saddle-node, Hopf, and Bogdanov–Takens types of bifurcations at coexisting equilibrium.

Table 1 Biological description of parameters

In the real world, the living region of prey and predator is inhomogeneous, and diffusion often occurs. Therefore, it is necessary to consider the spatial effect, such as reaction diffusion. Some work shows that space will affect the dynamic properties of the predator–prey model, such as spatial pattern, inhomogeneous periodic solution, etc. [710]. In addition, time delays often occur in predator–prey models, such as maturity delay and resource constraint delay. Time delays often cause spatial oscillations, such as periodic solutions [1114].

The resources in nature are limited, there will be competition within the population, and this competition is usually nonlocal. In [15, 16], the authors modified the \(\frac{u}{K}\) as \(\frac{1}{K} \int _{\Omega}G(x,y)u(y,t)\,dy\) to represent the nonlocal competition, where \(G(x,y)\) is some kernel function. In [17], Wu and Song studied a diffusive predator–prey model with nonlocal effect and delay, and suggested that steady-state, Hopf, and steady-state Hopf bifurcations may occur. In [18], Geng et al. studied Hopf, Turing, double-Hopf, and Turing–Hopf bifurcations of a diffusive predator–prey model with nonlocal competition. In [1922], all the authors show that the nonlocal competition may induce stably spatially inhomogeneous bifurcating periodic solutions, which is different from the model without nonlocal competition. Inspired by the above work, we want to analyze the effect of nonlocal competition, time delay, and spatial diffusion on the model (1.1). Consider the following model

$$ \textstyle\begin{cases}\frac{\partial{u(x,t)}}{\partial{t}}=d_{1} \Delta u +u (1- \int _{\Omega}G(x,y)u(y,t)\,dy )- \frac{\alpha u v^{2}}{1 +uv}, \\ \frac{\partial{v(x,t)}}{\partial{t}}=d_{2}\Delta v +\beta ( \frac{ u(t-\tau ) v^{2}(t-\tau )}{1 + u(t-\tau )v(t-\tau )}- \gamma v ),\quad x\in \Omega , t>0 \\ \frac{\partial{u(x,t)}}{\partial{\bar{\nu}}}= \frac{\partial{v(x,t)}}{\partial{\bar{\nu}}}=0,\quad x\in \partial{ \Omega}, t>0 \\ u(x,\theta )=u_{0}(x,\theta )\geq 0,\qquad v(x,\theta )=v_{0}(x, \theta ) \geq 0, \quad x\in \bar{\Omega}, \theta \in [-\tau ,0]. \end{cases} $$
(1.4)

The biological descriptions of the parameters are given in Table 1. \(\int _{\Omega}G(x,y)u(y,t)\,dy\) represents the nonlocal competition effect.

The rest of this paper is organized as follows. In Sect. 2, we study the stability of coexisting equilibrium and the existence of a Hopf bifurcation. In Sect. 3, we study the property of a Hopf bifurcation. In Sect. 4, we give some numerical simulations to illustrate the theoretical results. In Sect. 5, we give a short conclusion.

2 Stability analysis

Choose \(\Omega =(0,l\pi )\), and the kernel function \(G(x,y)=\frac{1}{l\pi}\). Denote N as a positive integer set, and N 0 as a nonnegative integer set. \((0,0)\) and \((K,0)\) are boundary equilibria of system (1.4). The existence of positive equilibria of system (1.4) has been studied in [6], that is

Lemma 2.1

([6])

The existence of positive equilibria of system (1.4) can be divided into three cases:

\(\alpha >\alpha _{bt}:=\frac{4 (1 -\gamma )}{27 \gamma ^{2}}\): no positive equilibrium.

\(\alpha =\alpha _{bt}\) and \(\beta >\gamma \): one positive equilibrium \((\frac{2}{3}, \frac{3\gamma }{2(1 -\gamma )})\).

\(\alpha <\alpha _{bt}\) and \(\beta >\gamma \): two distinct equilibria \((u_{1}, v_{1})\) and \((u_{2}, v_{2})\), where \(u_{1}<\frac{2}{3}<u_{2}\), and \(u_{1,2}\) are two roots of \(u^{3}-u^{2}+\frac{\alpha \gamma ^{2}}{1 - \gamma }=0\), \(v_{1,2}=\frac{\gamma}{(1-\gamma )u_{1,2}}\).

2.1 Model with nonlocal competition

Make the following hypothesis

$$\begin{aligned}& \mathbf{(H_{0})}\qquad \alpha \leq \alpha _{bt}, \qquad \gamma < 1. \end{aligned}$$

If \({\mathbf{(H_{0})}}\) holds, then system (1.4) has one or two coexisting equilibria. Hereinafter, for brevity, we denote \(E_{*}(u_{*},v_{*})\) as the coexisting equilibrium. Linearize system (1.4) at \(E_{*}(u_{*},v_{*})\)

u t ( u ( x , t ) u ( x , t ) ) = D ( Δ u ( t ) Δ v ( t ) ) + L 1 ( u ( x , t ) v ( x , t ) ) + L 2 ( u ( x , t τ ) v ( x , t τ ) ) + L 3 ( u ˆ ( x , t ) v ˆ ( x , t ) ) ,
(2.1)

where

D = ( d 1 0 0 d 2 ) , L 1 = ( a 1 a 2 0 β γ ) , L 2 = ( 0 0 b 1 b 2 ) , L 3 = ( u 0 0 0 ) ,

and \(a_{1}=\frac{u_{*} v_{*}^{3} \alpha }{(1+u_{*} v_{*})^{2}}>0\), \(a_{2}=- \frac{u_{*} v_{*} (2+u_{*} v_{*}) \alpha }{(1+u_{*} v_{*})^{2}}<0\), \(b_{1}=\frac{v_{*}^{2} \beta }{(1+u_{*} v_{*})^{2}}>0\), \(b_{2}=\frac{u_{*} v_{*} (2+u_{*} v_{*}) \beta }{(1+u_{*} v_{*})^{2}}>0\), \(\hat{u}=\frac{1}{l\pi}\int _{0}^{l\pi}u(y,t)\,dy\). The characteristic equations are

λ 2 + A n λ+ B n +( C n b 2 λ) e λ τ =0,n N 0 ,
(2.2)

where

A 0 = u a 1 + β γ , B 0 = ( u a 1 ) β γ , C 0 = b 2 ( u a 1 ) a 2 b 1 , A n = ( d 1 + d 2 ) n 2 l 2 + β γ a 1 , B n = d 1 d 2 n 4 l 4 + ( β γ d 1 a 1 d 2 ) n 2 l 2 a 1 β γ , C n = b 2 d 1 n 2 l 2 + a 1 b 2 a 2 b 1 , n N .
(2.3)

When \(\tau =0\), the characteristic equations (2.2) are

λ 2 +( A n b 2 )λ+ B n + C n =0,n N 0 ,
(2.4)

where

{ A 0 b 2 = a 1 + u b 2 , B 0 + C 0 = ( a 1 u ) ( b 2 β γ ) a 2 b 1 , A n b 2 = n 2 l 2 ( d 1 + d 2 ) a 1 + β γ b 2 , B n + C n = d 1 d 2 n 4 l 4 + n 2 l 2 [ d 2 a 1 + d 1 ( β γ b 2 ) ] B n + C n = + a 1 ( b 2 β γ ) a 2 b 1 , n N .
(2.5)

Make the following hypothesis

( H 1 ) A n b 2 >0, B n + C n >0,for n N 0 .

Theorem 2.1

For system (1.4), assume \(\tau =0\) and \({\mathbf{(H_{0})}}\) hold. Then, \(E_{*}(u_{*},v_{*})\) is locally asymptotically stable under \({\mathbf{(H_{1})}}\).

Proof

If \({\mathbf{(H_{1})}}\) holds, we can obtain that the characteristic roots of (2.4) all have negative real parts. Then, \(E_{*}(u_{*},v_{*})\) is locally asymptotically stable. □

Let iω (\(\omega >0\)) be a solution of Eq. (2.2), then

$$ - \omega ^{2}+\text{i} \omega A_{n}+B_{n}+(C_{n}-b_{2} \text{i} \omega ) ( \text{cos} \omega \tau -\text{i} \text{sin} \omega \tau ) =0.$$

We can obtain \(\text{cos}\omega \tau = \frac{\omega ^{2} ( b_{2} A_{n}+C_{n})-B_{n} C_{n}}{C_{n}^{2}+d^{2} \omega ^{2}}\), \(\text{sin}\omega \tau = \frac{\omega (A_{n} C_{n}+B_{n} b_{2}-b_{2} \omega ^{2} )}{C_{n}^{2}+b_{2}^{2} \omega ^{2}}\). This leads to

$$ \omega ^{4}+\omega ^{2} \bigl(A_{n}^{2}-2 B_{n}-b_{2}^{2} \bigr)+B_{n}^{2}-C_{n}^{2}=0. $$
(2.6)

Let \(z = \omega ^{2}\), then (2.6) becomes

$$ z^{2}+z \bigl(A_{n}^{2}-2 B_{n}-b_{2}^{2} \bigr)+B_{n}^{2}-C_{n}^{2}=0, $$
(2.7)

and the roots of (2.7) are \(z^{\pm}=\frac{1}{2}[-P_{n} \pm \sqrt{P_{n}^{2}-4Q_{n}R_{n}}] \), where \(P_{n}=A_{k}^{2}-2 B_{k}-b_{2}^{2}\), \(Q_{n}=B_{n}+C_{n}\), and \(R_{n}=B_{n}-C_{n}\). If \({\mathbf{(H_{0})}}\) and \({\mathbf{(H_{1})}}\) hold, \(Q_{n}>0\) (n N 0 ). By direct calculation, we have

P 0 = ( a 1 u ) 2 + β 2 γ 2 b 2 2 , P k = ( a 1 d 1 k 2 l 2 ) 2 + ( d 2 n 2 l 2 + β γ ) 2 b 2 2 , R 0 = a 2 b 1 ( a 1 u ) ( b 2 + β γ ) R k = d 1 d 2 k 4 l 4 + ( b 2 d 1 a 1 d 2 + d 1 β γ ) k 2 l 2 + a 2 b 1 a 1 b 2 a 1 β γ , for  k N .

Define

W 1 = { n | R n < 0 , n N 0 } , W 2 = { n | R n > 0 , P n < 0 , P n 2 4 Q n R n > 0 , n N 0 } , W 3 = { n | R n > 0 , P n 2 4 Q n R n < 0 , n N 0 } ,

and

$$ \begin{aligned} &\omega ^{\pm}_{n}= \sqrt{z_{n}^{\pm}}, \\ &\tau ^{j,\pm}_{n}= \textstyle\begin{cases} \frac{1}{\omega ^{\pm}_{n}} \arccos (V_{\cos}^{(n,\pm )})+2j \pi , & V_{\sin}^{(n,\pm )}\geq 0, \\ \frac{1}{\omega ^{\pm}_{n}} [2 \pi - \arccos (V_{\cos}^{(n, \pm )}) ]+2j \pi , & V_{\sin}^{(n,\pm )}< 0, \end{cases}\displaystyle \\ &V_{\cos}^{(n,\pm )}= \frac{(\omega ^{\pm}_{n}) ^{2} (b_{2} A_{n}+C_{n})-B_{n} C_{n}}{C_{n}^{2}+b_{2}^{2} (\omega ^{\pm}_{n}) ^{2}}, \\ & V_{\sin}^{(n, \pm )}= \frac{\omega ^{\pm}_{n} (A_{n} C_{n}+B_{n} b_{2}-b_{2} (\omega ^{\pm}_{n}) ^{2} )}{C_{n}^{2}+b_{2}^{2} (\omega ^{\pm}_{n})^{2}}. \end{aligned} $$
(2.8)

We have the following lemma.

Lemma 2.2

Assume \({\mathbf{(H_{0})}}\) and \({\mathbf{(H_{1})}}\) hold, then the following results hold.

Eq. (2.2) has a pair of purely imaginary roots \(\pm i\omega ^{+}_{n}\) at \(\tau ^{j,+}_{n}\) for j N 0 and n W 1 .

Eq. (2.2) has two pairs of purely imaginary roots \(\pm i\omega ^{\pm}_{n}\) at \(\tau ^{j,\pm}_{n}\) for j N 0 and n W 2 .

Eq. (2.2) has no purely imaginary root for n W 3 .

Lemma 2.3

Assume \({\mathbf{(H_{0})}}\) and \({\mathbf{(H_{1})}}\) hold. Then, \(\operatorname{Re}(\frac{d \lambda}{d \tau})|_{\tau =\tau ^{j,+}_{n}}>0\), \(\operatorname{Re} ( \frac{d \lambda}{d \tau})|_{\tau =\tau ^{j,-}_{n}}<0\) for n W 1 W 2 and j N 0 .

Proof

By (2.2), we have

$$ \biggl(\frac{d \lambda}{d \tau} \biggr)^{-1}= \frac{2 \lambda +A_{n}-b_{2} e^{-\lambda \tau}}{(C_{n}-b_{2}\lambda )\lambda e^{-\lambda \tau}}- \frac{\tau}{\lambda}. $$

Then,

$$ \begin{aligned} \biggl[\operatorname{Re} \biggl(\frac{d \lambda}{d \tau} \biggr)^{-1} \biggr]_{\tau =\tau ^{j, \pm}_{n}}&=\operatorname{Re} \biggl[ \frac{2 \lambda +A_{n}-b_{2}e^{-\lambda \tau}}{(C_{n}-b_{2}\lambda )\lambda e^{-\lambda \tau}}- \frac{\tau}{\lambda} \biggr]_{\tau =\tau ^{j,\pm}_{n}} \\ &= \biggl[\frac{1}{C_{n}^{2} +b_{2}^{2} \omega ^{2}} \bigl(2\omega ^{2}+A_{n}^{2}-2 B_{n}-b_{2}^{2} \bigr) \biggr]_{\tau =\tau ^{j,\pm}_{n}} \\ &=\pm \biggl[\frac{1}{C_{n}^{2} +b_{2}^{2} \omega ^{2}} \sqrt{ \bigl(A_{n}^{2}-2 B_{n}-b_{2}^{2} \bigr)^{2}-4 \bigl(B_{n}^{2}-C_{n}^{2} \bigr)} \biggr]_{\tau =\tau ^{j,\pm}_{n}}. \end{aligned} $$

Therefore, \(\operatorname{Re}(\frac{d \lambda}{d \tau})|_{\tau =\tau ^{j,+}_{n}}>0\), \(\operatorname{Re} ( \frac{d \lambda}{d \tau})|_{\tau =\tau ^{j,-}_{n}}<0\). □

Denote τ =min{ τ n 0 |n W 1 W 2 }. Note that \(\tau =\tau ^{j,+}_{m}(\tau =\tau ^{j,-}_{m})\) may be equal to \(\tau =\tau ^{j,+}_{n}(\tau =\tau ^{j,-}_{n})\), for some \(m\neq n\). In this case, high codimensional bifurcation will occur. In this paper, we do not consider this case. Then, we have the following theorem.

Theorem 2.2

Assume \({\mathbf{(H_{0})}}\) and \({\mathbf{(H_{1})}}\) hold, then the following statements are true for system (1.4).

\(E_{*}(u_{*},v_{*})\) is locally asymptotically stable for \(\tau >0\) when W 1 W 2 =.

\(E_{*}(u_{*},v_{*})\) is locally asymptotically stable for \(\tau \in [0,\tau _{*})\) when W 1 W 2 .

\(E_{*}(u_{*},v_{*})\) is unstable for \(\tau \in (\tau _{*},\tau _{*}+\varepsilon )\) for some \(\varepsilon >0\) when W 1 W 2 .

Hopf bifurcation occurs at \((u_{*},v_{*})\) when \(\tau =\tau ^{j,+}_{n}\) \((\tau =\tau ^{j,-}_{n})\), j N 0 , n W 1 W 2 .

2.2 Model without nonlocal competition

The model (1.4) without nonlocal competition is

$$ \textstyle\begin{cases} \frac{\partial{u(x,t)}}{\partial{t}}=d_{1} \Delta u +u (1-u )-\frac{\alpha uv^{2}}{1 +u v}, \\ \frac{\partial{v(x,t)}}{\partial{t}}=d_{2}\Delta v +\beta ( \frac{u(t-\tau )v^{2}(t-\tau )}{1 +u(t-\tau )v(t-\tau )}- \gamma v ), \quad x\in \Omega , t>0 \\ \frac{\partial{u(x,t)}}{\partial{\bar{\nu}}}= \frac{\partial{v(x,t)}}{\partial{\bar{\nu}}}=0,\quad x\in \partial{ \Omega}, t>0 \\ u(x,\theta )=u_{0}(x,\theta )\geq 0,\qquad v(x,\theta )=v_{0}(x, \theta ) \geq 0,\quad x\in \bar{\Omega}, \theta \in [-\tau ,0]. \end{cases} $$
(2.9)

Linearizing system (1.4) at \((u_{*},v_{*})\) gives:

( u t v t ) = d Δ ( u ( t ) v ( t ) ) + L 1 ( u ( t ) v ( t ) ) + L 2 ( u ( t τ ) v ( t τ ) ) ,
(2.10)

where

L 1 = ( a 1 u a 2 0 β γ ) , L 2 = ( 0 0 b 1 b 2 ) .

The characteristic equations of (2.10) are

λ 2 +λ A n + B n + ( C n b 2 λ ) e λ τ =0,n N 0 ,
(2.11)

where

A n = ( d 1 + d 2 ) n 2 l 2 + β γ + u a 1 , B n = d 1 d 2 n 4 l 4 + ( β γ d 1 + d 2 u a 1 d 2 ) n 2 l 2 + ( u a 1 ) β γ , C n = b 2 d 1 n 2 l 2 + ( u a 1 ) b 2 a 2 b 1 , n N 0 .

When \(\tau =0\), the characteristic Eq. (2.11) reduces to the following equation:

λ 2 + ( A n b 2 ) λ+ B n + C n =0,n N 0 ,
(2.12)

where

$$ \begin{aligned} \textstyle\begin{cases} A'_{n}-b_{2}=\frac{n^{2}}{l^{2}}(d_{1}+d_{2})-a_{1}+u_{*}+\beta \gamma -b_{2}, \\ B'_{n}+C'_{n}=d_{1}d_{2}\frac{n^{4}}{l^{4}}+\frac{n^{2}}{l^{2}} [d_{2} (u_{*}-a_{1})+d_{1}( \beta \gamma -b_{2}) ] \\ \hphantom{B'_{n}+C'_{n}=}{}+(a_{1}-u_{*}) (b_{2}- \beta \gamma )-a_{2} b_{1}. \end{cases}\displaystyle \end{aligned} $$
(2.13)

Make the following hypothesis

( H 2 ) A n b 2 > 0 , B n + C n > 0 , n N 0 .
(2.14)

Theorem 2.3

For system (2.9), assume \(\tau =0\) and \({\mathbf{(H_{0})}}\) holds. Then, \(E_{*}(u_{*},v_{*})\) is locally asymptotically stable under \({\mathbf{(H_{2})}}\).

Let iω (\(\omega >0\)) be a solution of Eq. (2.10), and \(z = \omega ^{2}\). Similarly, we can obtain \(z_{n,w}^{\pm}=\frac{1}{2}[-P'_{n} \pm \sqrt{(P'_{n})^{2}-4Q'_{n}R'_{n}}] \), where \(P'_{n}=(A'_{k})^{2}-2 B'_{k}-b_{2}^{2}\), \(Q'_{n}=B'_{n}+C'_{n}\), and \(R'_{n}=B'_{n}-C'_{n}\). If \({\mathbf{(H_{0})}}\) and \({\mathbf{(H_{2})}}\) hold, \(Q'_{n}>0\) (n N 0 ). By direct calculation, we have

P 0 = P 0 , R 0 = R 0 , P k = ( a 1 u d 1 k 2 l 2 ) 2 + ( d 2 n 2 l 2 + β γ ) 2 b 2 2 , R k = d 1 d 2 k 4 l 4 + [ d 2 ( u a 1 ) + d 1 ( b 2 + β γ ) ] k 2 l 2 + b 2 u + ( u a 1 ) β γ , for  k N .
(2.15)

Define

W 1 = { n | R n < 0 , n N 0 } , W 2 = { n | R n > 0 , P n < 0 , ( P n ) 2 4 Q n R n > 0 , n N 0 } , W 3 = { n | R n > 0 , ( P n ) 2 4 Q n R n < 0 , n N 0 } ,

and

$$ \begin{aligned} &\omega ^{\pm}_{n,w}= \sqrt{z_{n,w}^{\pm}}, \\ & \tau ^{j, \pm}_{n,w}=\textstyle\begin{cases} \frac{1}{\omega ^{\pm}_{n,w}} \arccos (V_{\cos,w}^{(n,\pm )})+2j \pi , & V_{\sin,w}^{(n,\pm )}\geq 0, \\ \frac{1}{\omega ^{\pm}_{n,w}} [2 \pi - \arccos (V_{\cos,w}^{(n, \pm )}) ]+2j \pi , & V_{\sin,w}^{(n,\pm )}< 0, \end{cases}\displaystyle \\ &V_{\cos,w}^{(n,\pm )}= \frac{(\omega ^{\pm}_{n,w}) ^{2} (b_{2} A'_{n}+C'_{n})-B'_{n} C'_{n}}{(C'_{n})^{2}+b_{2}^{2} (\omega ^{\pm}_{n,w}) ^{2}}, \\ & V_{\sin,w}^{(n, \pm )}= \frac{\omega ^{\pm}_{n,w} (A'_{n} C'_{n}+B'_{n} b_{2}-b_{2} (\omega ^{\pm}_{n,w}) ^{2} )}{(C'_{n})^{2}+b_{2}^{2} (\omega ^{\pm}_{n,w})^{2}}. \end{aligned} $$
(2.15)

We have the following lemma.

Lemma 2.4

Assume \({\mathbf{(H_{0})}}\) and \({\mathbf{(H_{1})}}\) hold, then the following results hold.

Eq. (2.11) has a pair of purely imaginary roots \(\pm i\omega ^{+}_{n,w}\) at \(\tau ^{j,+}_{n,w}\) for j N 0 and n W 1 .

Eq. (2.11) has two pairs of purely imaginary roots \(\pm i\omega ^{\pm}_{n,w}\) at \(\tau ^{j,\pm}_{n,w}\) for j N 0 and n W 2 .

Eq. (2.11) has no purely imaginary root for n W 3 .

Lemma 2.5

Assume \({\mathbf{(H_{0})}}\) and \({\mathbf{(H_{1})}}\) hold. Then, \(\operatorname{Re}(\frac{d \lambda}{d \tau})|_{\tau =\tau ^{j,+}_{n,w}}>0\), \(\operatorname{Re} ( \frac{d \lambda}{d \tau})|_{\tau =\tau ^{j,-}_{n,w}}<0\) for n W 1 W 2 and j N 0 .

Proof

By (2.11), we have

$$ \biggl(\frac{d \lambda}{d \tau} \biggr)^{-1}= \frac{2 \lambda +A'_{n}-b_{2} e^{-\lambda \tau}}{(C'_{n}-b_{2}\lambda )\lambda e^{-\lambda \tau}}- \frac{\tau}{\lambda}. $$

Then,

$$\begin{aligned}& \biggl[\operatorname{Re} \biggl(\frac{d \lambda}{d \tau} \biggr)^{-1} \biggr]_{\tau =\tau ^{j, \pm}_{n,w}} \\& \quad =\operatorname{Re} \biggl[ \frac{2 \lambda +A'_{n}-b_{2}e^{-\lambda \tau}}{(C'_{n}-b_{2}\lambda )\lambda e^{-\lambda \tau}}- \frac{\tau}{\lambda} \biggr]_{\tau =\tau ^{j,\pm}_{n,w}} \\& \quad = \biggl[\frac{1}{(C'_{n})^{2} +b_{2}^{2} \omega ^{2}} \bigl(2\omega ^{2}+ \bigl(A'_{n} \bigr)^{2}-2 B'_{n}-b_{2}^{2} \bigr) \biggr]_{\tau =\tau ^{j,\pm}_{n,w}} \\& \quad =\pm \biggl[\frac{1}{(C'_{n})^{2} +b_{2}^{2} \omega ^{2}} \sqrt{ \bigl( \bigl(A'_{n} \bigr)^{2}-2 B'_{n}-b_{2}^{2} \bigr)^{2}-4 \bigl( \bigl(B'_{n} \bigr)^{2}- \bigl(C'_{n} \bigr)^{2} \bigr)} \biggr]_{\tau =\tau ^{j, \pm}_{n,w}}. \end{aligned}$$

Therefore, \(\operatorname{Re}(\frac{d \lambda}{d \tau})|_{\tau =\tau ^{j,+}_{n,w}}>0\), \(\operatorname{Re} ( \frac{d \lambda}{d \tau})|_{\tau =\tau ^{j,-}_{n,w}}<0\). □

Denote τ =min{ τ n 0 |n W 1 W 2 }. We have the following theorem.

Theorem 2.4

Assume \({\mathbf{(H_{0})}}\) and \({\mathbf{(H_{1})}}\) hold, then the following statements are true for system (2.9).

\(E_{*}(u_{*},v_{*})\) is locally asymptotically stable for \(\tau >0\) when W 1 W 2 =.

\(E_{*}(u_{*},v_{*})\) is locally asymptotically stable for \(\tau \in [0,\tau '_{*})\) when W 1 W 2 .

\(E_{*}(u_{*},v_{*})\) is unstable for \(\tau \in (\tau '_{*},\tau _{*}+\varepsilon )\) for some \(\varepsilon >0\) when W 1 W 2 .

Hopf bifurcation occurs at \((u_{*},v_{*})\) when \(\tau =\tau ^{j,+}_{n,w}\) (\(\tau =\tau ^{j,-}_{n,w}\)), j N 0 , n W 1 W 2 .

3 Property of Hopf bifurcation

By the work [23, 24], we study the property of Hopf bifurcation. For fixed j N 0 and n W 1 W 2 , we denote \(\tilde{\tau}=\tau ^{j,\pm}_{n}\). Let \(\bar{u}(x,t)=u(x,\tau t)-u_{*}\) and \(\bar{v}(x,t)=v(x,\tau t)-v_{*}\). Dropping the bar, (1.4) can be written as

$$ \textstyle\begin{cases} \frac{\partial{u}}{\partial{t}}=\tau [d_{1} \Delta u +(u+u_{*}) (1-\frac{1}{l\pi} \int _{0}^{l\pi} (u(y,t)+u_{*} )\,dy )- \frac{\alpha (u+u_{*})(v+v_{*})^{2}}{1+(u+u_{*})(v+v_{*})} ], \\ \frac{\partial{v}}{\partial{t}}=\tau [d_{2} \Delta v + \frac{\beta (u(t-1)+u_{*})(v(t-1)+v_{*})^{2}}{1+(u(t-1)+u_{*})(v(t-1)+v_{*})}- \beta \gamma (v+v_{*}) ]. \end{cases} $$
(3.1)

We rewrite system (3.1) as the following system:

$$ \textstyle\begin{cases} \frac{\partial{u}}{\partial{t}}=\tau [d_{1} \Delta u +a_{1}u+a_{2}v-u_{*} \hat{u}+\alpha _{1}u^{2}-u\hat{u}+\alpha _{2}uv+ \alpha _{3} v^{2} \\ \hphantom{\frac{\partial{u}}{\partial{t}}=}{}+ \alpha _{4}u^{3}+\alpha _{5} u^{2}v+\alpha _{6} uv^{2} \\ \hphantom{ \frac{\partial{u}}{\partial{t}}=}{}+\alpha _{7} v^{3} ]+h.o.t., \\ \frac{\partial{v}}{\partial{t}}=\tau [d_{2} \Delta v +b_{1}u(t-1)+b_{2}v(t-1)- \beta \gamma v \\ \hphantom{\frac{\partial{v}}{\partial{t}}=}{} +\beta _{1}u^{2}(t-1)+\beta _{2}u(t-1)v(t-1)+\beta _{3} u^{2}(t-1) \\ \hphantom{\frac{\partial{v}}{\partial{t}}=}{}+\beta _{4} u^{3}(t-1)+\beta _{5}u^{2}(t-1)v(t-1) ] \\ \hphantom{\frac{\partial{v}}{\partial{t}}=}{}+\beta _{6}u(t-1)v^{2}(t-1)+ \beta _{7} v^{3}(t-1)]+h.o.t., \end{cases} $$
(3.2)

where \(\alpha _{1}=\frac{v_{*}^{3} \alpha }{(1+u_{*} v_{*})^{3}}\), \(\alpha _{2}=-\frac{2 v_{*} \alpha }{(1+u_{*} v_{*})^{3}}\), \(\alpha _{3}=-\frac{u_{*} \alpha }{(1+u_{*} v_{*})^{3}}\), \(\alpha _{4}=-\frac{v_{*}^{4} \alpha }{(1+u_{*} v_{*})^{4}}\), \(\alpha _{5}=\frac{v_{*}^{2} \alpha }{(1+u_{*} v_{*})^{4}}\), \(\alpha _{6}=\frac{(-1+2 u_{*} v_{*}) \alpha }{(1+u_{*} v_{*})^{4}}\), \(\alpha _{7}=\frac{u_{*}^{2} \alpha }{(1+u_{*} v_{*})^{4}}\), \(\beta _{1}=-\frac{v_{*}^{3} \beta }{(1+u_{*} v_{*})^{3}}\), \(\beta _{2}=\frac{2 v_{*} \beta }{(1+u_{*} v_{*})^{3}}\), \(\beta _{3}=\frac{u_{*} \beta }{(1+u_{*} v_{*})^{3}}\), \(\beta _{4}=\frac{v_{*}^{4} \beta }{(1+u_{*} v_{*})^{4}}\), \(\beta _{5}=-\frac{3 v_{*}^{2} \beta }{(1+u_{*} v_{*})^{4}}\), \(\beta _{6}=\frac{(1-2 u_{*} v_{*}) \beta }{(1+u_{*} v_{*})^{4}}\), \(\beta _{7}=-\frac{6 u_{*}^{2} \beta }{(1+u_{*} v_{*})^{4}}\).

Define the real-valued Sobolev space \(X:= \{(u,v)^{T}: u,v\in H^{2}(0,l\pi ), (u_{x},v_{x})|_{x=0,l \pi}=0 \} \), the complexification of X X C :=XiX={ x 1 +i x 2 | x 1 , x 2 X}, and the inner product \(\langle \tilde{u},\tilde{v}\rangle :=\int _{0}^{l\pi} \overline{u_{1}} v_{1}\,dx+ \int _{0}^{l\pi} \overline{u_{2}} v_{2}\,dx \) for \(\tilde{u}=(u_{1},u_{2})^{T}\), \(\tilde{v}=(v_{1},v_{2})^{T}\), u ˜ , v ˜ X C . The phase space \(\mathscr{C}:=C([-1,0],X)\) is with the sup norm, then we can write \(\phi _{t} \in \mathscr{C}\), \(\phi _{t}(\theta )=\phi (t+\theta )\) or \(-1\leq \theta \leq 0\). Denote \(\beta _{n}^{(1)}(x)=(\gamma _{n}(x),0)^{T}\), \(\beta _{n}^{(2)}(x)=(0,\gamma _{n}(x))^{T}\), and \(\beta _{n}=\{\beta _{n}^{(1)}(x),\beta _{n}^{(2)}(x) \}\), where \(\{\beta _{n}^{(i)}(x) \}\) is an orthonormal basis of X. We define the subspace of \(\mathscr{C}\) as B n :=span{ϕ(), β n ( j ) β n ( j ) |ϕC,j=1,2}, n N 0 . There exists a \(2\times 2\) matrix function \(\eta ^{n}(\sigma , \tilde{\tau})\) \(-1\le \sigma \le 0\), such that \(-\tilde{\tau} D\frac{n^{2}}{l^{2}} \phi (0)+\tilde{\tau}L(\phi )= \int _{-1}^{0}\,d\eta ^{n}(\sigma , \tau ) \phi (\sigma )\) for \(\phi \in \mathscr{C}\). The bilinear form on \(\mathscr{C}^{*} \times \mathscr{C}\) is defined by

$$ (\psi ,\phi )=\psi (0) \phi (0)- \int _{-1}^{0} \int _{\xi =0}^{\sigma} \psi (\xi -\sigma )\,d\eta ^{n}(\sigma , \tilde{\tau}) \phi (\xi )\,d\xi , $$
(3.3)

for \(\phi \in \mathscr{C}\), \(\psi \in \mathscr{C}^{*}\). Define \(\tau =\tilde{\tau}+\mu \), then the system undergoes a Hopf bifurcation at \((0,0)\) when \(\mu =0\), with a pair of purely imaginary roots \(\pm \text{i}\omega _{n_{0}}\). Let A denote the infinitesimal generators of the semigroup, and \(A^{*}\) be the formal adjoint of A under the bilinear form (3.3). Define the following function

δ ( n 0 ) = { 1 n 0 = 0 , 0 n 0 N .
(3.4)

Choose \(\eta _{n_{0}}(0,\tilde{\tau})=\tilde{\tau}[(-n_{0}^{2}/l^{2})D+L_{1}+L_{3} \delta (n_{n_{0}})]\), \(\eta _{n_{0}}(-1,\tilde{\tau})=-\tilde{\tau} L_{2}\), \(\eta _{n_{0}}(\sigma ,\tilde{\tau})=0\) for \(-1< \sigma <0\). Let \(p(\theta )=p(0)e^{\text{i}\omega _{n_{0}} \tilde{\tau} \theta}\) (\(\theta \in [-1,0]\)), \(q(\vartheta )=q(0)e^{-\text{i}\omega _{n_{0}} \tilde{\tau} \vartheta}\) (\(\vartheta \in [0,1]\)) be the eigenfunctions of \(A(\tilde{\tau})\) and \(A^{*}\) corresponds to \(i \omega _{n_{0}} \tilde{\tau}\), respectively. We can choose \(p(0)=(1,p_{1})^{T}\), \(q(0)=M(1,q_{2})\), where \(p_{1}=\frac{1}{a_{2}}(\text{i}\omega _{n_{0}}+d_{1} n_{0}^{2}/l^{2}-a_{1}+u_{*} \delta (n_{0}))\), \(q_{2}=a_{2}/(\text{i} \omega _{n_{0}}-b_{2} e^{\text{i} \tau \omega _{n_{0}} }+\beta \gamma +\frac{d_{2} n^{2}}{l^{2}})\), and \(M=(1+p_{1} q_{2}+\tilde{\tau}q_{2}(b_{1} +b_{2} p_{1})e^{-\text{i} \omega _{n_{0}} \tilde{\tau}})^{-1}\). Then, (3.1) can be rewritten in an abstract form

$$\begin{aligned} \frac{dU(t)}{dt} =&(\tilde{\tau}+\mu ) D\Delta U(t)+(\tilde{ \tau}+\mu ) \bigl[L_{1}(U_{t})+ L_{2} U(t-1) + L_{3} \hat{U}(t) \bigr] \\ &{}+F(U_{t},\hat{U}_{t},\mu ), \end{aligned}$$
(3.5)

where

F ( ϕ , μ ) = ( τ ˜ + μ ) ( α 1 ϕ 1 ( 0 ) 2 ϕ 1 ( 0 ) ϕ ˆ 1 ( 0 ) + α 2 ϕ 1 ( 0 ) ϕ 2 ( 0 ) + α 3 ϕ 2 ( 0 ) 2 + α 4 ϕ 1 3 ( 0 ) + α 5 ϕ 1 2 ( 0 ) ϕ 2 ( 0 ) + α 6 ϕ 1 ( 0 ) ϕ 2 2 ( 0 ) + α 7 ϕ 2 3 ( 0 ) β 1 ϕ 1 2 ( 1 ) + β 2 ϕ 1 ( 1 ) ϕ 2 ( 1 ) + β 3 ϕ 2 2 ( 1 ) + β 4 ϕ 1 3 ( 1 ) + β 4 ϕ 1 2 ( 1 ) ϕ 2 ( 1 ) + β 6 ϕ 1 ( 1 ) ϕ 2 2 ( 1 ) + β 7 ϕ 2 3 ( 1 ) ) ,
(3.6)

respectively, for \(\phi =(\phi _{1}, \phi _{2})^{T} \in \mathscr{C}\) and \(\hat{\phi}_{1}=\frac{1}{l\pi}\int _{0}^{l\pi}\phi \,dx\). Then, the space \(\mathscr{C}\) can be decomposed as \(\mathscr{C}=P\oplus Q\), where , \(Q=\{\phi \in \mathscr{C} | (q\gamma _{n_{0}}(x),\phi )=0 \text{ and } ( \bar{q}\gamma _{n_{0}}(x),\phi )=0 \}\). Then, system (3.6) can be rewritten as \(U_{t}=z(t)p(\cdot )\gamma _{n_{0}}(x) +\bar{z}(t)\bar{p}(\cdot ) \gamma _{n_{0}}(x)+ \omega (t, \cdot )\) and \(\hat{U_{t}}=\frac{1}{l\pi}\int _{0}^{l\pi} U_{t}\,dx\), where

$$ z(t)= \bigl(q\gamma _{n_{0}}(x),U_{t} \bigr),\qquad \omega (t, \theta )= U_{t}(\theta )-2 \operatorname{Re} \bigl\{ z(t)p(\theta ) \gamma _{n_{0}}(x) \bigr\} . $$
(3.7)

Then, we have \(\dot{z}(t)=\text{i}\omega ){n_{0}} \tilde{\tau} z(t)+\bar{q}(0)\langle F(0,U_{t}), \beta _{n_{0}}\rangle \). There exists a center manifold \(\mathcal{C}_{0}\) and ω can be written as follows near \((0,0)\):

$$ \omega (t,\theta )=\omega \bigl(z(t),\bar{z}(t),\theta \bigr)= \omega _{20}( \theta )\frac{z^{2}}{2} +\omega _{11}(\theta )z\bar{z}+\omega _{02}( \theta )\frac{\bar{z}^{2}}{2}+\cdots . $$
(3.8)

Then, restrict the system to the center manifold: \(\dot{z}(t)=\text{i}\omega _{n_{0}} \tilde{\tau} z(t)+g(z,\bar{z})\). Denote \(g(z,\bar{z})=g_{20}\frac{z^{2}}{2}+g_{11}z\bar{z} +g_{02} \frac{\bar{z}^{2}}{2}+g_{21}\frac{z^{2}\bar{z}}{2}+\cdots \) . By direct computation, we have

$$\begin{aligned}& g_{20}=2\tilde{\tau}M(\varsigma _{1}+q_{2} \varsigma _{2})I_{3}, \qquad g_{11}= \tilde{\tau}M(\varrho _{1}+q_{2} \varrho _{2})I_{3},\qquad g_{02}=\bar{g}_{20},\\& g_{21}=2\tilde{\tau}M \bigl[(\kappa _{11}+q_{2} \kappa _{21})I_{2} +(\kappa _{12}+q_{2} \kappa _{22})I_{4} \bigr], \end{aligned}$$

where \(I_{2}=\int _{0}^{l\pi} \gamma ^{2}_{n_{0}}(x)\,dx\), \(I_{3}=\int _{0}^{l\pi} \gamma ^{3}_{n_{0}}(x)\,dx\), \(I_{4}=\int _{0}^{l\pi} \gamma ^{4}_{n_{0}}(x)\,dx\), \(\varsigma _{1}=-\delta \mathrm{n}+\alpha _{1}+\alpha _{2} \xi + \alpha _{3} \xi ^{2}\), \(\varsigma _{2}=e^{-2 i \tau \omega _{n}} (\beta _{1}+\xi (\beta _{2}+ \beta _{3} \xi ))\), \(\varrho _{1}=\frac{1}{4} (2 \alpha _{1}-2 \delta \mathrm{n}+\alpha _{2} \bar{\xi} +\alpha _{2} \xi +2 \alpha _{3} \bar{\xi} \xi )\), \(\varrho _{2} =\frac{1}{4} (2 \beta _{1}+2 \beta _{3} \bar{\xi} \xi + \beta _{2} (\bar{\xi} +\xi ))\), \(\kappa _{11}=2 W_{11}^{(1)}(0) (-1+2 \alpha _{1}-\delta \mathrm{n}+ \alpha _{2} \xi )+2 W_{11}^{(2)}(0) (\alpha _{2}+2 \alpha _{3} \xi )+W_{20}^{(1)}(0) (-1+2 \alpha _{1}-\delta \mathrm{n}+\alpha _{2} \bar{\xi} )+W_{20}^{(2)}(0) (\alpha _{2}+2 \alpha _{3} \bar{\xi} )\), \(\kappa _{12}=\frac{1}{2} (3 \alpha _{4}+\alpha _{5} (\bar{\xi} +2 \xi )+\xi (2 \alpha _{6} \bar{\xi} +\alpha _{6} \xi +3 \alpha _{7} \bar{\xi} \xi ))\), \(\kappa _{21}=2 e^{-i \tau \omega _{n}} W_{11}^{(1)}(-1) (2 \beta _{1}+ \beta _{2} \xi )+2 e^{-i \tau \omega _{n}} W_{11}^{(2)}(-1) (\beta _{2}+2 \beta _{3} \xi )+e^{i \tau \omega _{n}} W_{20}^{(1)}(-1) (2 \beta _{1}+ \beta _{2} \bar{\xi} )+e^{i \tau \omega _{n}} W_{20}^{(2)}(-1) ( \beta _{2}+2 \beta _{3} \bar{\xi} )\), \(\kappa _{22}=\frac{1}{2} e^{-i \tau \omega _{n}} (3 \beta _{4}+ \beta _{5} (\bar{\xi} +2 \xi )+\xi (2 \beta _{6} \bar{\xi} +\beta _{6} \xi +3 \beta _{7} \bar{\xi} \xi ))\).

Now, we compute \(W_{20}(\theta )\) and \(W_{11}(\theta )\) for \(\theta \in [-1,0]\) to give \(g_{21}\). By (3.7), we have

$$ \dot{\omega}=\dot{U}_{t}-\dot{z} p \gamma _{n_{0}}(x)-\dot{\bar{z}} \bar{ p} \gamma _{n_{0}}(x) =A \omega +H(z,\bar{z},\theta ), $$
(3.9)

where

$$ H(z,\overline{z},\theta )=H_{20}(\theta ) \frac{z^{2}}{2}+H_{11} ( \theta )z \overline{z} +H_{02}( \theta ) \frac{\overline{z}^{2}}{2}+ \cdots . $$
(3.10)

Compared the coeffcients of (3.8) with (3.9), we have

$$ \begin{aligned} (A-2\text{i}\omega _{n_{0}} \tilde{\tau}I)\omega _{20}=-H_{20}( \theta ), \qquad A\omega _{11}(\theta )=-H_{11}(\theta ). \end{aligned} $$
(3.11)

Then, we have

$$ \begin{aligned} \omega _{20}(\theta )&= \frac{-g_{20}}{\text{i}\omega _{n_{0}} \tilde{\tau}}p(0)e^{\text{i} \omega _{n_{0}} \tilde{\tau} \theta} - \frac{\bar{g}_{02}}{3\text{i}\omega _{n_{0}} \tilde{\tau}}\bar{p}(0)e^{- \text{i}\omega _{n_{0}} \tilde{\tau} \theta} +E_{1}e^{2\text{i}\omega _{n_{0}} \tilde{\tau} \theta}, \\ \omega _{11}(\theta )&= \frac{g_{11}}{\text{i}\omega _{n_{0}} \tilde{\tau}}p(0)e^{\text{i} \omega _{n_{0}} \tilde{\tau} \theta} - \frac{\bar{g}_{11}}{\text{i}\omega _{n_{0}} \tilde{\tau}}\bar{p}(0)e^{- \text{i}\omega _{n_{0}} \tilde{\tau} \theta} +E_{2}, \end{aligned} $$
(3.12)

where \(E_{1}=\sum_{n=0}^{\infty}E_{1}^{(n)}\), \(E_{2}=\sum_{n=0}^{\infty}E_{2}^{(n)}\),

E 1 ( n ) = ( 2 i ω n 0 τ ˜ I 1 0 e 2 i ω n 0 τ ˜ θ d η n 0 ( θ , τ ¯ ) ) 1 F ˜ 20 , β n , E 2 ( n ) = ( 1 0 d η n 0 ( θ , τ ¯ ) ) 1 F ˜ 11 , β n , n N 0 , F ˜ 20 , β n = { 1 l π F ˆ 20 , n 0 0 , n = 0 , 1 2 l π F ˆ 20 , n 0 0 , n = 2 n 0 , 1 l π F ˆ 20 , n 0 = 0 , n = 0 , 0 , other , F ˜ 11 , β n = { 1 l π F ˆ 11 , n 0 0 , n = 0 , 1 2 l π F ˆ 11 , n 0 0 , n = 2 n 0 , 1 l π F ˆ 11 , n 0 = 0 , n = 0 , 0 , other ,
(3.13)

and \(\hat{F}_{20}=2(\varsigma _{1},\varsigma _{2})^{T}\), \(\hat{F}_{11}=2(\varrho _{1},\varrho _{2})^{T}\).

Thus, we can obtain

$$ \begin{aligned} c&_{1}(0)=\frac{\text{i}}{2\omega _{n} \tilde{\tau}} \biggl(g_{20}g_{11}-2 \vert g_{11} \vert ^{2}- \frac{ \vert g_{02} \vert ^{2}}{3} \biggr) +\frac{1}{2}g_{21},\qquad \mu _{2}=- \frac{\operatorname{Re}(c_{1}(0))}{\operatorname{Re}(\lambda ' (\tilde{\tau}))}, \\ T&_{2}=-\frac{1}{\omega _{n_{0}} \tilde{\tau}} \bigl[\operatorname{Im} \bigl(c_{1}(0) \bigr)+ \mu _{2} \operatorname{Im} \bigl(\lambda ' \bigl(\tau ^{j}_{n} \bigr) \bigr) \bigr], \qquad \beta _{2}= 2 \operatorname{Re} \bigl(c_{1}(0) \bigr). \end{aligned} $$
(3.14)

Theorem 3.1

For any critical value \(\tau ^{j}_{n}\) (nS, j N 0 ), we have the following results:

When \(\mu _{2}>0\) (resp., <0), the Hopf bifurcation is forward (resp., backward).

When \(\beta _{2}<0\) (resp., >0), the bifurcating periodic solutions on the center manifold are orbitally asymptotically stable (resp., unstable).

When \(T_{2}>0\) (resp., \(T_{2}<0\)), the period increases (resp., decreases).

4 Numerical simulations

To study the effect of nonlocal competition, we also give numerical simulations for models (1.4) and (2.9). Fix the following parameters:

$$ \alpha =1.2, \qquad \gamma =0.25,\qquad d_{1}=0.1,\qquad d_{2}=0.3,\qquad l=0.8. $$

System (1.4) has two positive equilibria \(E_{1}\approx (0.4126,0.8079)\) and \(E_{2}\approx (0.8670,0.3845)\). Since \(E_{2}\) is always unstable, we mainly analyze the stability of \(E_{1}\). We can obtain the bifurcation diagrams of systems (1.4) and (2.9) with β (Fig. 1), where \(\beta _{1}\approx 0.1929\) and \((\beta _{2},\tau _{2} )\approx (0.2064,80.1787)\). We also compute some parameters for model (1.4) with different β (Table 2).

Figure 1
figure 1

Bifurcation diagram of system (1.4) for β and τ. (Left: model (1.4). Right: model (2.9))

Table 2 Some parameters for model (1.4) with different ξ

From Fig. 1, we can see that increasing β is no benefit to the stability of coexisting equilibrium. For the model (2.9), a spatially inhomogeneous periodic solution curve does not exist. For the model (1.4), when \(0<\beta <\beta _{1}\), the stability of the coexistence equilibrium \(E_{1}\) is similar to model (2.9). When \(\beta >\beta _{1}\), the spatially inhomogeneous periodic solution curve (\(\tau _{1}^{0}\)) exists, and is larger than the spatially homogeneous periodic solution curve (\(\tau _{0}^{0}\)) for \(\beta _{1}<\beta <\beta _{2}\). This means that the spatially homogeneous periodic solution will appear first, and the spatially inhomogeneous periodic solution is usually unstable. However, when \(\beta >\beta _{2}\), the spatially inhomogeneous periodic solution curve (\(\tau _{1}^{0}\)) is smaller than the spatially homogeneous periodic solution curve (\(\tau _{0}^{0}\)). This means that the spatially inhomogeneous periodic solution will appear first, and may be asymptotically stable.

Choose \(\beta =0.18\), when \(\tau < \tau _{*}\approx 89.9073\), the coexistence equilibrium \(E_{1}\) is asymptotically stable for models (1.4) and (2.9) (Fig. 2). When \(\tau >\tau _{*}\), the coexistence equilibrium \(E_{1}\) is unstable and the spatial homogeneous periodic solution appears for models (1.4) and (2.9) (Fig. 3).

Figure 2
figure 2

Numerical simulations for prey population with \(\beta =0.18\) \(\tau =22\). (Left: model (1.4). Right: model (2.9))

Figure 3
figure 3

Numerical simulations for prey population with \(\beta =0.18\) \(\tau =22\). (Left: model (1.4). Right: model (2.9))

Choose \(\beta =0.5\), when \(\tau < \tau _{*}\approx 10.3066\), the coexistence equilibrium \(E_{1}\) is asymptotically stable for models (1.4) and (2.9) (Fig. 4). However, when \(\tau _{*}< \tau < \tau _{0}^{0} \approx 19.7545\), for model (1.4) the coexistence equilibrium \(E_{1}\) is unstable and the spatial homogeneous periodic solution does not exist. The stably spatial inhomogeneous periodic solution appears (Fig. 5 left). However, for the model (2.9), the coexistence equilibrium \(E_{1}\) is still asymptotically stable (Fig. 5 right). When \(\tau >\tau _{0}^{0}\), for model (1.4), the coexistence equilibrium \(E_{1}\) is unstable and the unstably spatial homogeneous periodic solution exists. The stably spatial inhomogeneous periodic solution still exists (Fig. 6 left). However, for the model (2.9), the coexistence equilibrium \((u_{*}, v_{*})\) is unstable, and the stably spatial homogeneous periodic solution appears (Fig. 6 right).

Figure 4
figure 4

Numerical simulations for prey population with \(\beta =0.5\) \(\tau =5\). (Left: model (1.4). Right: model (2.9))

Figure 5
figure 5

Numerical simulations for prey population with \(\beta =0.5\) \(\tau =12\). (Left: model (1.4). Right: model (2.9))

Figure 6
figure 6

Numerical simulations for prey population with \(\beta =0.5\) \(\tau =22\). (Left: model (1.4). Right: model (2.9))

5 Conclusion

In this paper, we study a delayed diffusive predator–prey model with nonlocal competition in prey and schooling behavior among predators. We mainly study the local stability of coexisting equilibrium and the existence of Hopf bifurcation. We also studied the property of bifurcating periodic solutions by the normal form method and center manifold theorem. Our results show that diffusion and delay can induce a spatially inhomogeneous periodic solution, which is usually unstable. However, the model incorporating nonlocal competition may have a stably spatially inhomogeneous periodic solution.