When the first Lyapunov coefficient is equal to 0, the Hopf bifurcation may degenerate and Bautin bifurcation occurs. In this section, Bautin bifurcation is investigated at positive equilibrium \({{E}_{2}}\) with τ and r as bifurcation parameters. Next, according to the research in [22–24], the properties of Hopf bifurcation and Bautin bifurcation are studied. Let \((\tau _{0},r_{0})\) be the Bautin bifurcation point for bifurcation analysis. After scaling \(t\to \frac{t}{\tau }\) and introducing \(\mu =\tau -{{\tau }_{0}}\), \(\varsigma =r-{{r}_{0}}\), \({{u}_{1}}=x(t)-{{x}_{2}}\), \({{u}_{2}}(t)=y(t)-{{y}_{2}}\), system (1) can be written as
$$ \textstyle\begin{cases} {{{\dot{u}}}_{1}}(t)= ( {{\tau }_{0}}+\mu ) \{ {{{a}_{12}}{{u}_{2}}(t)-d{{u}_{1}}(t-1)} \\ \hphantom{{{{\dot{u}}}_{1}}(t)=}{}-{{a}_{13}}u_{2}^{2}(t)+{{a}_{14}}u_{2}^{3}(t)+h.o.t \} , \\ {{{\dot{u}}}_{2}}(t)= ( {{\tau }_{0}}+\mu ) \{ {{{a}_{21}}{{u}_{1}}(t)+a_{22}^{*}{{u}_{2}}(t)+{{a}_{23}}u_{2}^{2}(t)} \\ \hphantom{{{{\dot{u}}}_{2}}(t)=}{}-\beta {{u}_{1}}(t){{u}_{2}}(t)-{{a}_{24}}u_{2}^{3}(t) \} , \end{cases} $$
(11)
where
$$\begin{aligned}& {{a}_{13}}=\frac{\alpha b}{{{ ( {{y}_{2}}+b )}^{3}}}, \qquad {{a}_{14}}= \frac{\alpha b}{{{ ( {{y}_{2}}+b )}^{4}}}, \\& a_{22}^{*}=({{r}_{0}}+\varsigma ) \biggl[ \biggl( 1-\frac{2y_{2}}{k} \biggr) (y_{2}-m)+ \biggl( y_{2}- \frac{{{y_{2}^{2}}}}{k} \biggr) \biggr]-\beta {{x}_{2}}, \\& {{a}_{23}}=({{r}_{0}}+\varsigma ) \biggl[ \biggl( 1+ \frac{m}{k} \biggr)-\frac{3}{k}{{y}_{2}} \biggr],\qquad {{a}_{24}}= \frac{{{r}_{0}}+\varsigma }{k}. \end{aligned}$$
The above system (11) is transformed into a functional differential equation on the phase space \(C=C ( [-1 ,0],{{\mathbb{R}}^{2}} )\)
$$\begin{aligned} \dot{u}(t)=L(\mu ,\varsigma ){{u}_{t}}+F(\mu ,\varsigma ,{{u}_{t}}), \end{aligned}$$
where \(u(t)={{ ( {{u}_{1}}(t),{{u}_{2}}(t) )}^{T}}\in {{ \mathbb{R}}^{2}}\), \(L:C\to {{\mathbb{R}}^{2}}\), \({{u}_{t}}=u(t+\theta )\), \(\theta \in [-1,0]\), and \(F:{{\mathbb{R}}^{2}}\times C\to {{\mathbb{R}}^{2}}\). For \(\varphi (\theta )= ( {{\varphi }_{1}}(\theta ),{{\varphi }_{2}}( \theta ) )\in C\), we have
$$\begin{aligned} L(\mu ,\varsigma )\varphi ={{B}_{1}}\varphi (0)+{{B}_{2}} \varphi (-1). \end{aligned}$$
where
and
By the Riesz representation theorem, there exists bounded variation functions \(\eta (\theta ,\mu ,\varsigma )\) in \(\theta \in [-1,0]\) such that
$$\begin{aligned} L(\mu ,\varsigma )\varphi = \int _{-1}^{0}{d\eta (\theta ,\mu , \varsigma ) \varphi (\theta )},\quad \text{for } \varphi \in C[-1,0]. \end{aligned}$$
In fact, \(\eta (\theta ,\mu ,\varsigma )\) can be
$$\begin{aligned} d\eta (\theta ,\mu ,\varsigma )={{B}_{1}}\delta (\theta )+{{B}_{2}} \delta (\theta +1), \end{aligned}$$
where δ is the Dirac delta function. Now define operator
$$\begin{aligned} A(\mu ,\varsigma )\varphi =& \textstyle\begin{cases} \frac{d\varphi (\theta )}{d\theta }, &\text{for } \theta \in [ -1,0), \\ \int _{-1}^{0}{d\eta (\theta ,\mu ,\varsigma )\varphi (\theta )},& \text{for } \theta =0, \end{cases}\displaystyle \end{aligned}$$
and
$$\begin{aligned} R(\mu ,\varsigma )\varphi =& \textstyle\begin{cases} 0, &\text{for } \theta \in [ -1,0), \\ F(\mu ,\varsigma ,\theta ),& \text{for } \theta =0. \end{cases}\displaystyle \end{aligned}$$
System (11) is equivalent to the following abstract ordinary differential equation:
$$\begin{aligned} {{\dot{u}}_{t}}=A(\mu ){{u}_{t}}+R(\mu ){{u}_{t}}, \end{aligned}$$
where \({{u}_{t}}=u(t+\theta )\), \(\theta \in [-1,0]\). The adjoint operator \({{A}^{*}}\) of A is
$$\begin{aligned} {{A}^{*}}(\mu ,\varsigma )\psi =& \textstyle\begin{cases}- \frac{d\psi (s)}{ds},& \text{for } s\in (0,1], \\ \int _{-1}^{0}\psi (-s)\,d\eta (s,\mu ,\varsigma ),& \text{for } s=0. \end{cases}\displaystyle \end{aligned}$$
For \(\varphi \in C[-1,0]\) and \(\psi \in C[0,1]\), define the bilinear form
$$\begin{aligned} \bigl\langle \psi (s),\varphi (\theta ) \bigr\rangle =\bar{\psi }(0) \varphi (0)- \int _{-1}^{0}{ \int _{0}^{\theta }{\bar{\psi }(\xi - \theta )\,d\eta ( \theta )\varphi (\xi )\,d\xi }}, \end{aligned}$$
where \(\eta (\theta )=\eta (\theta ,0,0)\). It is easy to show \(\langle \psi ,A\varphi \rangle = \langle {{A}^{*}} \psi ,\varphi \rangle \). Let \(q(\theta )\) be the eigenvector corresponding to eigenvalue \(i{{\omega }_{0}}\) of \(A(0,0)\) and \({{q}^{*}}(s)\) be the eigenvector corresponding to eigenvalue \(-i{{\omega }_{0}}\) of \({{A}^{*}}\), namely
$$\begin{aligned} A(0,0)q(\theta )=i{{\omega }_{0}}q(\theta ), \qquad {{A}^{*}} {{q}^{*}}(s)=-i{{ \omega }_{0}} {{q}^{*}}(s). \end{aligned}$$
By simple calculation, we have
$$\begin{aligned} q(\theta )={{(1,{{q}_{1}})}^{T}} {{e}^{i{{\omega }_{0}}\theta }}, \qquad {{q}^{*}}(s)=D{{ \bigl(1,q_{1}^{*} \bigr)}^{T}} {{e}^{i{{ \omega }_{0}}s},} \end{aligned}$$
where
$$\begin{aligned}& {{q}_{1}}= \frac{i{{\omega }_{0}}-d{{e}^{i{{\omega }_{0}}{{\tau }_{0}}}}}{{{a}_{12}}},\qquad q_{1}^{*}= \frac{-i{{\omega }_{0}}-d{{e}^{-i{{\omega }_{0}}{{\tau }_{0}}}}}{{{a}_{12}}}, \\& \bar{D}= \frac{1}{1+\bar{q}_{1}^{*}{{q}_{1}}-d{{\tau }_{0}}{{e}^{-i{{\omega }_{0}}{{\tau }_{0}}}}}. \end{aligned}$$
Defining \(z(t)= \langle {{q}^{*}},{{u}_{t}} \rangle \) and \({{W}^{(\mu ,\varsigma )}}(t,\theta )={{u}_{t}}(\theta )-2 \operatorname{Re}[z(t)q(\theta )]\), on the center manifold \({{C}_{0}}\), we have
$$\begin{aligned} {{W}^{(\mu ,\varsigma )}}(t,\theta )={{W}^{(\mu ,\varsigma )}} \bigl(z(t), \bar{z}(t), \theta \bigr) \end{aligned}$$
and
$$\begin{aligned} {{W}^{(\mu ,\varsigma )}} \bigl(z(t),\bar{z}(t),\theta \bigr) =&W_{20}^{(\mu , \varsigma )}(\theta )\frac{{{z}^{2}}}{2}+W_{11}^{(\mu ,\varsigma )}( \theta )z\bar{z} \\ &{}+W_{02}^{(\mu ,\varsigma )}(\theta )\frac{{{{\bar{z}}}^{2}}}{2}+W_{30}^{( \mu ,\varsigma )}( \theta )\frac{{{z}^{3}}}{6}+\cdots. \end{aligned}$$
For convenience, rewriting \(W_{ij}^{(\mu ,\varsigma )}\) as \({{W}_{ij}}\), this is \(W_{ij}^{(\mu ,\varsigma )}=(W_{ij}^{(1)},W_{ij}^{(2)})\), \(i+j\ge 2\), we have
$$\begin{aligned} \dot{z}(t) =& \bigl\langle {{q}^{*}},{{{ \dot{u}}}_{t}} \bigr\rangle = \bigl\langle {{q}^{*}},A{{u}_{t}}+R{{u}_{t}} \bigr\rangle \\ =& \bigl\langle {{A}^{*}} {{q}^{*}},{{u}_{t}} \bigr\rangle + \bigl\langle {{q}^{*}},R{{u}_{t}} \bigr\rangle \\ =&i{{\omega }_{0}}z+{{{\bar{q}}}^{*}}(0)F \bigl(0,{{W}^{(\mu , \varsigma )}}(t,\theta )+2\operatorname{Re} \bigl[z(t)q(0) \bigr] \bigr) \\ =&i{{\omega }_{0}}z+g(z,\bar{z}), \end{aligned}$$
where
$$ g(z,\bar{z})=\sum_{i+j\ge 2}{{{g}_{ij}} \frac{1}{i!j!}} {{z}^{i}} {{ \bar{z}}^{j}}. $$
(12)
Moreover, one obtains
(13)
The expression of \({{u}_{t}}(\theta )={{ ( {{u}_{1t}}(\theta ),{{u}_{2t}}(\theta ) )}^{T}}\) is easy to get
$$ \textstyle\begin{cases} {{u}_{1}}(0)=z+\bar{z}+W_{20}^{(1)}(0)\frac{{{z}^{2}}}{2} \\ \hphantom{{{u}_{1}}(0)=}{} +W_{11}^{(1)}(0)z\bar{z}+W_{02}^{(1)}(0) \frac{{{{\bar{z}}}^{2}}}{2}+\cdots, \\ {{u}_{2}}(0)={{q}_{1}}z+{{{\bar{q}}}_{1}}\bar{z}+W_{20}^{(2)}(0) \frac{{{z}^{2}}}{2} \\ \hphantom{{{u}_{2}}(0)=}{} +W_{11}^{(2)}(0)z\bar{z}+W_{02}^{(2)}(0) \frac{{{{\bar{z}}}^{2}}}{2}+\cdots. \end{cases} $$
(14)
Substituting (14) into (13), then comparing the coefficient with (12), we can get
$$\begin{aligned}& {{g}_{21}}=2 ( {{M}_{1}} {{G}_{210}}-{{M}_{2}} {{G}_{211}} ),\qquad {{g}_{02}}=2 \bigl({{M}_{1}} \bar{q}_{1}^{2}-{{M}_{2}} {{\bar{q}}_{1}} \bigr), \\& {{g}_{20}}=2 \bigl({{M}_{1}}q_{1}^{2}-{{M}_{2}} {{q}_{1}} \bigr), \\& {{g}_{11}}={{M}_{1}} {{q}_{1}} {{ \bar{q}}_{1}}-\beta \bar{q}_{1}^{*}D ( {{q}_{1}}+{{{\bar{q}}}_{1}} ) ( {{\tau }_{0}}+ \mu ),\qquad {{g}_{12}}=2 ( {{M}_{1}} {{G}_{120}}-{{M}_{2}} {{G}_{121}} ), \\& {{g}_{40}}=12 ( 2{{M}_{1}} {{G}_{400}}-{{M}_{2}} {{G}_{401}} ), \qquad {{g}_{31}}=3 ( 2{{M}_{1}} {{G}_{310}}-{{M}_{2}} {{G}_{311}} ), \\& {{g}_{13}}=3 ( 2{{M}_{1}} {{G}_{130}}-{{M}_{2}} {{G}_{131}} ),\qquad {{g}_{32}}=6 ( 2{{M}_{1}} {{G}_{320}}-{{M}_{2}} {{G}_{321}} ), \\& {{g}_{22}}=4 ( {{M}_{1}} {{G}_{220}}-{{M}_{2}} {{G}_{221}} ), \\& {{g}_{03}}=6{{M}_{1}} {{ \bar{q}}_{1}}W_{02}^{(2)}-6{{M}_{2}} \biggl( \frac{{{{\bar{q}}}_{1}}W_{02}^{(1)}(0)}{2}+W_{02}^{(2)}(0) \biggr), \\& {{g}_{30}}=6{{M}_{1}} {{q}_{1}}W_{20}^{(2)}(0)-6{{M}_{2}} \biggl( \frac{{{q}_{1}}W_{20}^{(1)}(0)}{2}+W_{20}^{(2)}(0) \biggr), \end{aligned}$$
where
$$\begin{aligned}& {{M}_{2}}=\beta \bar{q}_{1}^{*}D ( {{ \tau }_{0}}+\mu ), \qquad {{M}_{1}}= ( {{\tau }_{0}}+\mu ) \bigl[ D({{a}_{14}}-{{a}_{13}})+D \bar{q}_{1}^{*}({{a}_{23}}-{{a}_{24}}) \bigr], \\& {{G}_{210}}={{\bar{q}}_{1}}W_{20}^{(2)}(0)+2{{q}_{1}}W_{11}^{(2)}(0), \qquad {{G}_{120}}={{q}_{1}}W_{02}^{(2)}(0)+2{{ \bar{q}}_{1}}W_{11}^{(2)}(0), \\& {{G}_{400}}=\frac{{{q}_{1}}W_{30}^{(2)}}{3}+ \frac{{{ ( W_{20}^{(2)} )}^{2}}}{4},\qquad {{G}_{310}}={{q}_{1}}W_{21}^{(2)}(0)+ \frac{{{{\bar{q}}}_{1}}W_{30}^{(2)}(0)}{3}, \\& {{G}_{130}}={{\bar{q}}_{1}}W_{12}^{(2)}(0)+ \frac{{{q}_{1}}W_{03}^{(2)}(0)}{3},\qquad {{G}_{221}}=W_{11}^{(1)}(0)W_{11}^{(2)}(0)+ \frac{{{H}_{2}}}{2}, \\& {{G}_{311}}= \frac{{{{\bar{q}}}_{1}}W_{30}^{(1)}(0)+W_{30}^{(2)}(0)}{3}+{{q}_{1}}W_{21}^{(1)} (0)+W_{21}^{(2)}(0), \\& {{G}_{401}}=\frac{{{q}_{1}}W_{30}^{(1)}(0)+W_{30}^{(2)}(0)}{3}+ \frac{W_{20}^{(1)}(0)W_{20}^{(2)}(0)}{2}, \\& {{G}_{121}}= \frac{{{{\bar{q}}}_{1}}W_{02}^{(1)}(0)+W_{02}^{(2)}(0)}{2}+{{\bar{q}}_{1}}W_{11}^{(1)}(0)+W_{11}^{(2)}(0), \\& {{G}_{211}}= \frac{{{{\bar{q}}}_{1}}W_{20}^{(1)}(0)+W_{20}^{(2)}(0)}{2}+{{q}_{1}}W_{11}^{(1)}(0)+W_{11}^{(2)}(0), \\& {{G}_{131}}=\frac{{{q}_{1}}W_{03}^{(1)}(0)+W_{03}^{(2)}(0)}{3}+{{ \bar{q}}_{1}}W_{12}^{(1)}(0)+W_{12}^{(2)}(0), \\& {{G}_{320}}=W_{21}^{(2)}(0)W_{11}^{(2)}(0)+ \frac{2{{{\bar{q}}}_{1}}W_{31}^{(2)}(0)+W_{30}^{(2)}(0)W_{02}^{(2)}(0)}{6}, \\& {{G}_{321}}=W_{21}^{(1)}(0)W_{11}^{(2)}(0)+W_{21}^{(2)}(0)W_{11}^{(1)}(0)+ \frac{{{{\bar{q}}}_{1}}W_{31}^{(1)}(0)+W_{31}^{(2)}(0)}{3}+ \frac{{{H}_{1}}}{6}, \\& {{G}_{220}}={{\bar{q}}_{1}}W_{21}^{(2)}(0)+{{q}_{1}}W_{12}^{(2)}(0)+{{ \bigl( W_{11}^{(2)}(0) \bigr)}^{2}}+ \frac{W_{20}^{(2)}(0)W_{02}^{(2)}(0)}{2}, \end{aligned}$$
and
$$\begin{aligned}& {{H}_{1}}=W_{30}^{(1)}(0)W_{02}^{(2)}(0)+W_{30}^{(2)}(0)W_{02}^{(1)}(0), \\& {{H}_{2}}={{\bar{q}}_{1}}W_{21}^{(1)}(0)+{{q}_{1}}W_{12}^{(1)}(0)+W_{21}^{(2)}(0)+W_{12}^{(2)}(0) \\& \hphantom{{{H}_{2}}=}+{}W_{20}^{(1)}(0)W_{02}^{(2)}(0)+W_{02}^{(1)}(0)W_{20}^{(2)}(0). \end{aligned}$$
Furthermore, the calculation results show that
$$\begin{aligned} {{\mu }_{2}}=- \frac{\operatorname{Re} [ {{C}_{1}}(0) ]}{\operatorname{Re} [ {\lambda }'({{\tau }_{0}}) ]},\qquad {{ \beta }_{2}}=2 \operatorname{Re} {{C}_{1}}(0), \end{aligned}$$
where
$$\begin{aligned} {{C}_{1}}(0)=\frac{i}{2{{\omega }_{0}}} \biggl( {{g}_{20}} {{g}_{11}}-2{{ \vert {{g}_{11}} \vert }^{2}}- \frac{1}{3}{{ \vert {{g}_{02}} \vert }^{2}} \biggr)+\frac{{{g}_{21}}}{2}. \end{aligned}$$
The sign of \({{\mu }_{2}}\) reveals the direction of the Hopf bifurcation and \({{\beta }_{2}}\) the stability of periodic solutions, so we have the following.
Theorem 3.1
If \({{\mu }_{2}}>0\) (\({{\mu }_{2}}<0\)), then the Hopf bifurcation is supercritical(subcritical) and the bifurcating periodic solutions on the center manifold are stable(unstable) if \({{\beta }_{2}}<0\) (\({{\beta }_{2}}>0\)).
Next, to illustrate the existence of Bautin bifurcation, we need to calculate the first Lyapunov coefficient and the second Lyapunov coefficient as follows:
$$\begin{aligned} {{l}_{1}}(\mu ,\varsigma )=\frac{1}{2{{\omega }_{0}}} \biggl[ \operatorname{Re} {{g}_{21}}-\frac{1}{{{\omega }_{0}}} \operatorname{Im}({{g}_{20}} {{g}_{11}}) \biggr] \end{aligned}$$
and
$$\begin{aligned} 12{{l}_{2}}(0,0) =&\frac{1}{{{\omega }_{0}}} \operatorname{Re} {{g}_{32}}+ \frac{1}{\omega _{0}^{2}}\operatorname{Im} \biggl[ {{{g}_{20}} {{{ \bar{g}}}_{31}}-{{g}_{11}}(4{{g}_{31}}+3{{{ \bar{g}}}_{22}})} {- \frac{1}{3}{{g}_{02}}(g{}_{40}+{{{ \bar{g}}}_{13}})-{{g}_{30}} {{g}_{12}}} \biggr] \\ &{} +\frac{1}{\omega _{0}^{3}}\operatorname{Re} \biggl[ {{{g}_{20}} {{{ \bar{g}}}_{11}}(3{{g}_{12}}-{{{\bar{g}}}_{30}})+{{g}_{20}} {{g}_{02}} \biggl( {{{\bar{g}}}_{12}} -\frac{1}{3}{{g}_{30}} \biggr)} \\ &{} + \frac{1}{3}{{g}_{20}} {{{\bar{g}}}_{02}}g{}_{30}+{{g}_{11}} {{{\bar{g}}}_{02}} \biggl( \frac{5}{3}{{{\bar{g}}}_{30}}+3{{g}_{12}} \biggr) \biggr] \\ &{} +\frac{1}{\omega _{0}^{3}}\operatorname{Re} \biggl[ \frac{1}{3}{{g}_{11}} {{g}_{02}} {{{ \bar{g}}}_{03}}-4g_{2}^{11}{{g}_{30}} \biggr]+ \frac{3}{\omega _{0}^{3}}\operatorname{Im}({{g}_{20}} {{g}_{11}}) \operatorname{Im} {{g}_{21}} \\ &{} +\frac{1}{\omega _{0}^{4}}\operatorname{Im} \bigl[ {{g}_{11}} {{{ \bar{g}}}_{02}}(\bar{g}_{20}^{2}-3{{{ \bar{g}}}_{20}} {{g}_{11}}-4g_{11}^{2} \bigr] \\ &{}+\frac{1}{\omega _{0}^{4}}\operatorname{Im}({{g}_{20}} {{g}_{11}}) \bigl[ 3\operatorname{Re}({{g}_{20}} {{g}_{11}})-2{{ \vert {{g}_{02}} \vert }^{2}} \bigr]. \end{aligned}$$
For the transversality condition, obviously, if \(\frac{\partial {{l}_{1}}(\mu ,\varsigma )}{\partial \varsigma }\ne 0\), then the map \((\mu ,\varsigma )\to ( \frac{\xi (\mu )}{\omega (\mu )}, {{l}_{1}}( \mu ,\varsigma ) )\) at \((\mu ,\varsigma )=(0,0)\) is
Theorem 3.2
If \({{l}_{1}}(0,0)=0\), \(\frac{\partial {{l}_{1}}(\mu ,\varsigma )}{\partial \varsigma }\ne 0\) and \({{l}_{2}}(0,0)\ne 0\) are all satisfied, then system (1) undergoes Bautin bifurcation at the critical point \(({{\tau }_{0}},{{r}_{0}})\).