1 Introduction

Over the years, dynamics analysis of economic systems has mainly focused on the stability behavior of the system equilibrium point. According to the theory of nonlinear dynamical systems, the phenomena of fluctuations over time are not stochastic influences arising from external factors, but because of the nonlinear relationship between the variables of the economic system. Market share attraction models are used for analyzing Interbrand competitive structures. These models have received more and more attention [1,2,3,4,5].

Recently, many research papers suggested that the mathematical model of economic system dynamics is more realistic and appropriate when it is modeled by discrete-time equations. The dynamics of the discrete-time models can exhibit much richer dynamics than those observed in continuous-time counterparts and can lead to chaotic behaviors [6,7,8,9,10,11,12,13,14,15,16,17,18,19,20].

A market share attraction model is as follows:

$$ \textstyle\begin{cases} x_{t + 1} = x_{t} + \lambda_{1}x_{t}(Bs_{1t} - x_{t}), \\ y_{t + 1} = y_{t} + \lambda_{2}y_{t}(Bs_{2t} - y_{t}), \end{cases} $$
(1.1)

where \(x_{t}\) and \(y_{t}\) denote the marketing efforts of the two brands respectively. B denotes the total sales potential of the market. \(s_{1t} = x_{t}^{\beta_{1}} / (x_{t}^{\beta_{1}} + ky_{t}^{\beta_{2}})\), \(s_{2t} = ky_{t}^{\beta_{2}} / (x_{t}^{\beta_{1}} + ky_{t}^{\beta_{2}})\). The parameters \(\beta_{1}\) and \(\beta_{2}\) denote the elasticities of the attraction of firm (or brand) i with regard to the effort of firm i. The parameter k denotes the relative effectiveness ratio of the effort made by the firms. The parameters \(\lambda_{1}\) and \(\lambda_{2}\) measure the rate of adjustment. From system (1.1), the mapping form is obtained:

$$ \textstyle\begin{cases} x' \to x + \lambda_{1}x(B\frac{x^{\beta_{1}}}{x^{\beta_{1}} + ky^{\beta_{2}}} - x), \\ y' \to y + \lambda_{2}y(B\frac{ky^{\beta_{2}}}{x^{\beta_{1}} + ky^{\beta_{2}}} - y), \end{cases} $$
(1.2)

where \(\lambda_{1}\), \(\lambda_{2}\), \(\beta_{1}\), \(\beta_{2}\), k, and B are real and positive parameters. We consider only the values of the exponents \(\beta_{1}\) and \(\beta_{2}\) at the interval \((0,1)\) since empirical studies show that realistic values are in this range.

Our objectives are to study the dynamical behaviors of system (1.2). Sufficient conditions for the existence of flip bifurcation are derived by using the bifurcation theory and the center manifold theorem. Moreover, system (1.2) shows a rich variety of nonlinear dynamics, including bifurcations and chaos.

The paper is organized as follows. In Sect. 2, the stability and existence of the fixed points of system (1.2) are discussed. In Sect. 3, the existence of flip bifurcation is obtained by using the center manifold theorem and bifurcation theory. Numerical simulations are illustrated to confirm the theoretical results in Sect. 4. Some conclusions are presented in Sect. 5.

2 The existence and stability of the fixed points

We study the existence of fixed points. Also, we investigate the stability properties of system (1.2). The fixed points of map (1.2) are the solutions of the following equations:

$$ \textstyle\begin{cases} \lambda_{1}x(B\frac{x^{\beta_{1}}}{x^{\beta_{1}} + ky^{\beta_{2}}} - x) = 0, \\ \lambda_{2}y(B\frac{ky^{\beta_{2}}}{x^{\beta_{1}} + ky^{\beta_{2}}} - y) = 0. \end{cases} $$
(2.1)

For all parameter values, equation (2.1) has three solutions \(O(0,0)\), \(P_{1}(B,0)\), and \(P_{2}(0,B)\). As the map does not define O, it is not a fixed point. \(P_{1}\) and \(P_{2}\) are the fixed points of the map.

There is an interior fixed point \(E(x,y)\) of map (1.2), which is the solution of the following system:

$$ \textstyle\begin{cases} B\frac{x^{\beta_{1}}}{x^{\beta_{1}} + ky^{\beta_{2}}} - x = 0, \\ B\frac{ky^{\beta_{2}}}{x^{\beta_{1}} + ky^{\beta_{2}}} - y = 0. \end{cases} $$
(2.2)

From equation (2.2), we have

$$ G(x) = k^{1 / (1 - \beta_{2})}x^{(1 - \beta_{1}) / (1 - \beta_{2})} + x - B = 0. $$
(2.3)

G is a continuous function, \(G(B) > 0\), \(G(0) < 0\), and \(G'(x) > 0\) for \(x > 0\), so there is a unique positive solution, \(x^{ *} \in (0,B)\), the fixed point is \(E(x^{ *},B - x^{ *} )\).

A particularly simple solution is obtained in the case \(\beta_{1} = \beta_{2}\), \(x^{ *} = B / (1 + k^{1 / (1 - \beta_{2})})\).

We will study the local stability of the fixed points.

The Jacobian matrix of system (1.2) at \((x,y)\) is given as follows:

J(x,y)= [ 1 + λ 1 a 1 λ 1 b 1 λ 2 a 2 1 + λ 2 b 2 ] ,
(2.4)

where

$$\begin{aligned}& a_{1} = \biggl( \frac{Bx^{2\beta_{1}} + Bk(\beta_{1} + 1)x^{\beta_{1}}y^{\beta_{2}}}{(x^{\beta_{1}} + ky^{\beta_{2}})^{2}} - 2x \biggr), \\& b_{1} = \frac{Bk\beta_{2}y^{\beta_{2} - 1}x^{\beta_{1} + 1}}{(x^{\beta_{1}} + ky^{\beta_{2}})^{2}}, \\& a_{2} = \frac{Bk\beta_{1}y^{\beta_{2} + 1}x^{\beta_{1} - 1}}{(x^{\beta_{1}} + ky^{\beta_{2}})^{2}}, \\& b_{2} = \biggl( \frac{Bk^{2}y^{2\beta_{2}} + Bk(\beta_{2} + 1)x^{\beta_{1}}y^{\beta_{2}}}{(x^{\beta_{1}} + ky^{\beta_{2}})^{2}} - 2y \biggr). \end{aligned}$$

So the characteristic equation of the Jacobian matrix J can be written as

$$ s^{2} - (2 + \lambda_{1}a_{1} + \lambda_{2}b_{2})s + \bigl(1 + \lambda_{1}a_{1} + \lambda_{2}b_{2} + \lambda_{1} \lambda_{2}(a_{1}b_{2} - a_{2}b_{1}) \bigr) = 0. $$
(2.5)

In order to study the stability at the positive fixed point, we use the following lemmas, which can be easily proved by the relations between roots and coefficients of the quadratic equation.

Let \(F(s) = s^{2} + Ms + N\) be the characteristic equation of eigenvalues associated with the Jacobian matrix evaluated at a fixed point \((x^{*},y^{*})\). Let \(s _{1}\) and \(s _{2}\) be the two roots of \(F(s)\), M and N be coefficients of the quadratic equation.

Lemma 2.1

([21])

We have the following definitions for \((x^{*},y^{*})\):

  1. (1)

    \((x^{*},y^{*})\) is called a sink if \(\vert s_{1} \vert < 1\) and \(\vert s_{2} \vert < 1\), so the sink is locally asymptotically stable;

  2. (2)

    \((x^{*},y^{*})\) is called a source if \(|s_{1}| > 1\) and \(\vert s_{2} \vert > 1\), so the source is locally unstable;

  3. (3)

    \((x^{*},y^{*})\) is called a saddle if \(|s_{1}| > 1\) and \(|s_{2}| < 1\) (or \(|s_{1}| < 1\) and \(|s_{2}| > 1\));

  4. (4)

    \((x^{*},y^{*})\) is non-hyperbolic if either \(|s_{1}| = 1\) or \(|s_{2}| = 1\).

Lemma 2.2

([21])

Let \(F(s) = s^{2} + Ms + N\). Suppose that \(F(1) > 0\), \(s_{1}\) and \(s_{2}\) are two roots of \(F(s) = 0\). Then

  1. (1)

    \(\vert s_{1} \vert < 1\) and \(\vert s_{2} \vert < 1\) if and only if \(F( - 1) < 0\), \(N < 1\);

  2. (2)

    \(\vert s_{1} \vert < 1\) and \(\vert s_{2} \vert > 1\) (or \(\vert s_{1} \vert > 1\) and \(\vert s_{2} \vert < 1\)) if and only if \(F( - 1) < 0\);

  3. (3)

    \(\vert s_{1} \vert > 1\) and \(\vert s_{2} \vert > 1\) if and only if \(F(1) > 0\), \(N > 1\);

  4. (4)

    \(s_{1} = - 1\) and \(\vert s_{2} \vert \ne 1\) if and only if \(F( - 1) = 0\) and \(M \ne 0,2\);

  5. (5)

    \(s_{1}\) and \(s_{2}\) are complex and \(\vert s_{1} \vert = \vert s_{2} \vert = 1\) if and only if \(M^{2} - 4N < 0\) and \(N = 1\).

Now we state the following three propositions.

Proposition 1

The eigenvalues of \(J(B,0)\) are \(s_{1} = 1 - \lambda_{1}B\) and \(s_{2} = 1\), then \((B,0)\) is non-hyperbolic.

Proposition 2

The eigenvalues of \(J(0,B)\) are \(s_{1} = 1\) and \(s_{2} = 1 - \lambda_{1}B\), then \((0,B)\) is non-hyperbolic.

For the fixed point \(O(0,0)\), we can get a solution in the case of identical firms.

Here we consider the symmetric case of identical firms obtained for

$$\lambda_{1} = \lambda_{2} = \lambda > 0,\qquad \beta_{1} = \beta_{2} = \beta > 0, \quad k = 1. $$

Note that this steady state allocation belongs to the diagonal \(\Delta = \{ (x,y)|x = y\}\).

For the symmetric map, the Jacobian matrix, computed at a point of the diagonal Δ, is

J(x,x)= [ 1 2 λ x + λ B ( β + 2 ) 4 λ B β 4 λ B β 4 1 2 λ x + λ B ( β + 2 ) 4 ] .
(2.6)

The eigenvalues are

$$s_{1} = 1 + \frac{1}{2}\lambda B - 2\lambda x,\qquad s_{2} = 1 + \frac{1}{2}\lambda B(1 + \beta ) - 2\lambda x. $$

The fixed point \(O(0, 0)\) has the following topological properties:

$$s_{1} = 1 + \frac{1}{2}\lambda B, \qquad s_{2} = 1 + \frac{1}{2}\lambda B(1 + \beta ). $$

Proposition 3

The eigenvalues of \(J(0,0)\) are \(s_{1} = 1 + \frac{1}{2}\lambda B > 1\) and \(s_{2} = 1 + \frac{1}{2}\lambda B(1 + \beta ) > 1\), then \((0,0)\) is a source; the source is locally unstable.

For the fixed point \(E(x^{ *},B - x^{ *} )\), \(x^{ *} \in (0,B)\), we can get a special solution in the case of identical firms \(E(\frac{B}{2},\frac{B}{2})\).

Here we consider the symmetric case of identical firms obtained for

$$\lambda_{1} = \lambda_{2} = \lambda, \qquad \beta_{1} = \beta_{2} = \beta. $$

\(J(x,y)\) evaluated at the interior fixed point

J(x,y)= [ 1 + λ a 1 λ b 1 λ a 2 1 + λ b 2 ] ,
(2.7)

where

$$\begin{aligned}& a_{1} = \biggl( \frac{Bx^{2\beta} + Bk(\beta + 1)x^{\beta} y^{\beta}}{ (x^{\beta} + ky^{\beta} )^{2}} - 2x \biggr), \qquad b_{1} = \frac{Bk\beta y^{\beta - 1}x^{\beta + 1}}{(x^{\beta} + ky^{\beta} )^{2}}, \\& a_{2} = \frac{Bk\beta y^{\beta + 1}x^{\beta - 1}}{(x^{\beta} + ky^{\beta} )^{2}}, \qquad b_{2} = \biggl( \frac{Bk^{2}y^{2\beta} + Bk(\beta + 1)x^{\beta} y^{\beta}}{ (x^{\beta} + ky^{\beta} )^{2}} - 2y \biggr). \end{aligned}$$

The characteristic equation of (2.7) evaluated at the positive fixed point \(E(x^{*},y^{*})\) can be written as

$$ s^{2} - (2 + G\lambda )s + \bigl(1 + G\lambda + H\lambda^{2} \bigr) = 0, $$
(2.8)

where \(G = a_{1} + b_{2}\), \(H = a_{1}b_{2} - a_{2}b_{1}\).

Let \(F(s) = s^{2} - (2 + G\lambda )s + (1 + G\lambda + H\lambda^{2})\).

Then \(F(1) = H\lambda^{2}\), \(F( - 1) = H\lambda^{2} + 2G\lambda + 4\).

Using Lemma 2.1, we obtain the local dynamics of the fixed point \(E(x^{ *},y^{ *} )\).

Proposition 4

Let \(E(x^{ *},y^{ *} )\) be the positive fixed point of Eq. (1.2);

  1. 1.

    E is a sink if one of the following conditions holds:

    1. (a)

      \(- 2\sqrt{H} \le G < 0\) and \(0 < \lambda < - \frac{G}{H}\).

    2. (b)

      \(G < - 2\sqrt{H}\) and \(0 < \lambda < \frac{ - G - \sqrt{G^{2} - 4H}}{ H}\).

    So E locally asymptotically stable.

  2. 2.

    E is a source if one of the following conditions holds:

    1. (a)

      \(- 2\sqrt{H} \le G < 0\) and \(\lambda > - \frac{G}{H}\).

    2. (b)

      \(G < - 2\sqrt{H}\) and \(\lambda > \frac{ - G + \sqrt{G^{2} - 4H}}{H}\).

    3. (c)

      \(G \ge 0\).

  3. 3.

    E is a saddle if the following condition holds:

    $$G < - 2\sqrt{H} \quad \textit{and}\quad \frac{ - G - \sqrt{G^{2} - 4H}}{H} < \lambda < \frac{ - G + \sqrt{G^{2} - 4H}}{H}. $$
  4. 4.

    E is non-hyperbolic if one of the following conditions holds:

    1. (a)

      \(G < - 2\sqrt{H}\) and \(\lambda = \frac{ - G \pm \sqrt{G^{2} - 4H}}{H}\) and \(\lambda \ne - \frac{2}{G}, - \frac{4}{G}\).

    2. (b)

      \(- 2\sqrt{H} < G < 0\) and \(\lambda = - \frac{G}{H}\).

3 Bifurcation analysis

In this section, we discuss the flip bifurcation in system (1.2) at the positive fixed point \(E(x^{ *},y^{ *} )\). We choose parameter λ as a bifurcation parameter to study the flip bifurcation of \(E(x^{ *},y^{ *} )\) by using the center manifold theorem and the bifurcation theory [22,23,24,25].

Let \(F_{B1} = \{ ( B,k,\beta,\lambda ): \lambda = \frac{ - G + \sqrt{G^{2} - 4H}}{H}, G < - 2\sqrt{H},B,k,\beta,\lambda > 0 \}\), or \(F_{B2} = \{ ( B,k,\beta,\lambda ): \lambda = \frac{ - G - \sqrt{G^{2} - 4H}}{H}, G < - 2\sqrt{H},B,k,\beta,\lambda > 0 \}\).

Let \(H_{B} = \{ ( B,k,\beta,\lambda ): \lambda = - \frac{G}{H}, - 2\sqrt{H} < G < 0,B,k,\beta,\lambda > 0 \}\).

The fixed point \((x^{ *},y^{ *} )\) can undergo a flip bifurcation when parameters vary in a small neighborhood of \(F_{B1}\) or \(F_{B2}\), and the Neimark–Sacker bifurcation of \(E(x^{ *},y^{ *} )\) if parameters vary in a small neighborhood of \(H_{B}\).

3.1 Flip bifurcation analysis

We will discuss the flip bifurcation of (1.2) at \(E(x^{ *},y^{ *} )\) when parameters vary in the small neighborhood of \(F_{B1}\). Similar arguments can be applied to the other case \(F_{B2}\). Taking parameters \(( B,k,\beta,\lambda_{1} )\) arbitrarily from \(F_{B1}\), we consider system (1.2) with \(( B,k,\beta,\lambda_{1} )\), which is described by

$$ \textstyle\begin{cases} x' \to x + \lambda_{1}x(B\frac{x^{\beta}}{x^{\beta} + ky^{\beta}} - x), \\ y' \to y + \lambda_{1}y(B\frac{ky^{\beta}}{x^{\beta} + ky^{\beta}} - y). \end{cases} $$
(3.1)

Map (3.1) has a unique positive fixed point \(E(x^{ *},y^{ *} )\), whose eigenvalues are \(s_{1} = - 1\), \(s_{2} = 3 + G\lambda_{1}\) with \(\vert s_{2} \vert \ne 1\) by Proposition 4, where \(x^{ *} = B / (1 + k^{1 / (1 - \beta )})\), \(y^{ *} = B - [B / (1 + k^{1 / (1 - \beta )})]\).

Choosing \(\lambda^{*}\) as a bifurcation parameter, we consider a perturbation of (3.1) as follows:

$$ \textstyle\begin{cases} x' \to x + (\lambda_{1} + \lambda^{*})x(B\frac{x^{\beta}}{x^{\beta} + ky^{\beta}} - x), \\ y' \to y + (\lambda_{1} + \lambda^{*})y(B\frac{ky^{\beta}}{x^{\beta} + ky^{\beta}} - y), \end{cases} $$
(3.2)

where \(\vert \lambda^{*} \vert \ll 1\), which is a small perturbation parameter.

Let \(u = x - x^{*}\) and \(v = y - y^{*}\). Then we transform the fixed point \(E(x^{ *},y^{ *} )\) of map (3.2) into the origin. We have

( u v ) ( a 11 u + a 12 v + a 13 u 2 + a 14 u v + a 15 v 2 + e 1 u 3 + e 2 u 2 v + e 3 v 2 u + e 4 v 3 + b 1 u λ + b 2 v λ + b 3 u 2 λ + b 4 u v λ + b 5 v 2 λ + O ( ( | u | + | v | + | λ | ) 4 ) a 21 u + a 22 v + a 23 u 2 + a 24 u v + a 25 v 2 + d 1 u 3 + d 2 u 2 v + d 3 v 2 u + d 4 v 3 + c 1 u λ + c 2 v λ + c 3 u 2 λ + c 4 u v λ + c 5 v 2 λ + O ( ( | u | + | v | + | λ | ) 4 ) ) ,
(3.3)

where

$$\begin{aligned}& a_{11} = 1 + \lambda_{1} \biggl( \frac{Bx^{*2\beta} + Bk(\beta + 1)x^{*\beta} y^{*\beta}}{(x^{*\beta} + ky^{*\beta} )^{2}} - 2x^{*} \biggr), \qquad a_{12} = - \frac{\lambda_{1}Bk\beta y^{*\beta - 1}x^{*\beta + 1}}{(x^{*\beta} + ky^{*\beta} )^{2}}, \\& a_{13} = \frac{\lambda_{1}Bk\beta x^{*\beta - 1}y^{*\beta} [(1 - \beta )x^{*\beta} + k(1 + \beta )y^{*\beta} ]}{(x^{*\beta} + ky^{*\beta} )^{3}} - 2, \\& a_{14} = \lambda_{1}kB\beta x^{*\beta} y^{*\beta - 1}\frac{(\beta - 1)x^{*\beta} - k(\beta + 1)y^{*\beta}}{(x^{*\beta} + ky^{*\beta} )^{3}}, \\& a_{15} = \lambda_{1}kB\beta y^{*\beta - 2}x^{*\beta + 1} \frac{(1 - \beta )x^{*\beta} + k(\beta + 1)y^{*\beta}}{(x^{*\beta} + ky^{*\beta} )^{3}}, \\& b_{1} = \biggl( \frac{Bx^{*2\beta} + Bk(\beta + 1)x^{*\beta} y^{*\beta}}{(x^{*\beta} + ky^{*\beta} )^{2}} - 2x^{*} \biggr), \\& b_{2} = - \frac{Bk\beta y^{*\beta - 1}x^{*\beta + 1}}{(x^{*\beta} + ky^{*\beta} )^{2}}, \\& b_{3} = \frac{Bk\beta x^{*\beta - 1}y^{*\beta} [(1 - \beta )x^{*\beta} + k(1 + \beta )y^{*\beta} ]}{(x^{*\beta} + ky^{*\beta} )^{3}} - 2, \\& b_{4} = kB\beta x^{*\beta} y^{*\beta - 1}\frac{(\beta - 1)x^{*\beta} - k(\beta + 1)y^{*\beta}}{(x^{*\beta} + ky^{*\beta} )^{3}}, \\& b_{5} = kB\beta y^{*\beta - 2}x^{*\beta + 1}\frac{(1 - \beta )x^{*\beta} + (k\beta + 1)y^{*\beta}}{(x^{*\beta} + ky^{*\beta} )^{3}}, \\& a_{21} = - \frac{\lambda_{1}Bk\beta y^{*\beta + 1}x^{*\beta - 1}}{(x^{*\beta} + ky^{*\beta} )^{2}}, \\& a_{22} = 1 + \lambda_{1} \biggl( \frac{Bk^{2}y^{*2\beta} + Bk(\beta + 1)x^{*\beta} y^{*\beta}}{(x^{*\beta} + ky^{*\beta} )^{2}} - 2y^{*} \biggr), \\& a_{23} = \lambda_{1}kB\beta x^{*\beta - 2}y^{*\beta + 1} \frac{(1 - \beta )ky^{*\beta} + (\beta + 1)x^{*\beta}}{ (x^{*\beta} + ky^{*\beta} )^{3}}, \\& a_{24} = \lambda_{1}kB\beta x^{*\beta - 2}y^{*\beta + 1} \frac{(\beta - 1)ky^{*\beta} - (\beta + 1)x^{*\beta}}{(x^{*\beta} + ky^{*\beta} )^{3}}, \\& a_{25} = \lambda_{1}kB\beta x^{*\beta} y^{*\beta - 1}\frac{(1 - \beta )ky^{*\beta} + (\beta + 1)x^{*\beta}}{(x^{*\beta} + ky^{*\beta} )^{3}} - 2, \\& c_{1} = - \frac{Bk\beta y^{*\beta + 1}x^{*\beta - 1}}{(x^{*\beta} + ky^{*\beta} )^{2}}, \\& c_{2} = \biggl( \frac{Bk^{2}y^{*2\beta} + Bk(\beta + 1)x^{*\beta} y^{*\beta}}{(x^{*\beta} + ky^{*\beta} )^{2}} - 2y^{*} \biggr), \\& c_{3} = kB\beta x^{*\beta - 2}y^{*\beta + 1}\frac{(1 - \beta )ky^{*\beta} + (\beta + 1)x^{*\beta}}{(x^{*\beta} + ky^{*\beta} )^{3}}, \\& c_{4} = kB\beta x^{*\beta - 2}y^{*\beta + 1}\frac{(\beta - 1)ky^{*\beta} - (\beta + 1)x^{*\beta}}{(x^{*\beta} + ky^{*\beta} )^{3}}, \\& c_{5} = kB\beta x^{*\beta} y^{*\beta - 1}\frac{(1 - \beta )ky^{*\beta} + (\beta + 1)x^{*\beta}}{ (x^{*\beta} + ky^{*\beta} )^{3}} - 2, \\& d_{1} = \frac{\lambda_{1}Bk\beta y^{*\beta + 1}x^{*\beta - 3}[( - 3\beta - 2 - \beta^{2})x^{*2\beta} + 4k(\beta^{2} - 1)x^{*\beta} y^{*\beta} + k^{2}(3\beta - \beta^{2} - 2)y^{*2\beta} ]}{(x^{*\beta} + ky^{*\beta} )^{4}}, \\& d_{2} = \frac{\lambda_{1}Bk\beta x^{*\beta - 2}y^{*\beta} [k^{2}(\beta - 1)^{2}y^{*2\beta} + (\beta + 1)^{2}x^{*2\beta} + 2k(1 - 2\beta^{2})x^{*\beta} y^{*\beta} ]}{(x^{*\beta} + ky^{*\beta} )^{4}}, \\& d_{3} = \frac{\lambda_{1}Bk\beta^{2}x^{*\beta - 1}y^{*\beta - 1}[ - (1 + \beta )x^{*2\beta} + k^{2}(1 - \beta )y^{*2\beta} ) + 4k\beta x^{*\beta} y^{*\beta} ]}{(x^{*\beta} + ky^{*\beta} )^{4}}, \\& d_{4} = \frac{\lambda_{1}Bk\beta y^{*\beta - 2}x^{*\beta} [(\beta^{2} - 1)(k^{2}y^{*2\beta} + x^{*2\beta} ) - 2k(1 + 2\beta^{2})x^{*\beta} y^{*\beta} ]}{(x^{*\beta} + ky^{*\beta} )^{4}}. \\& e_{1} = \frac{\lambda_{1}Bk\beta x^{*\beta - 2}y^{*\beta} [(\beta^{2} - 1)(x^{*2\beta} + k^{2}y^{*2\beta} ) - 2k(1 + 2\beta^{2})x^{*\beta} y^{*\beta} ]}{(x^{*\beta} + ky^{*\beta} )^{4}}, \\& e_{2} = \frac{\lambda_{1}Bk\beta^{2}x^{*\beta - 1}y^{*\beta - 1}[(1 - \beta )x^{*2\beta} - k^{2}(1 + \beta )y^{*2\beta} ) + 4k\beta x^{*\beta} y^{*\beta} ]}{(x^{*\beta} + ky^{*\beta} )^{4}}, \\& e_{3} = \frac{\lambda_{1}Bk\beta x^{*\beta} y^{*\beta - 2}[2(2\beta - 1 - \beta^{2})x^{*2\beta} - k(1 + \beta^{2})x^{*\beta} y^{*\beta} + k^{2}(\beta + 1)^{2}y^{*2\beta} ]}{(x^{*\beta} + ky^{*\beta} )^{4}}, \\& e_{4} = \bigl(\lambda_{1}Bk\beta x^{*\beta + 1}y^{*\beta - 3} \bigl[\bigl(2\beta - 3 - \beta^{2}\bigr)x^{*2\beta} + 2k\bigl(2 \beta^{2} - \beta - 3\bigr)x^{*\beta} y^{*\beta} \\& \hphantom{e_{4} =}{}- k^{2}\bigl(\beta^{2} + 4\beta + 3 \bigr)y^{*2\beta} \bigr]\bigr)\big/\bigl(\bigl(x^{*\beta} + ky^{*\beta} \bigr)^{4}\bigr). \end{aligned}$$

Constructing an invertible matrix

T= ( a 12 a 12 1 a 11 s 2 a 11 )

and using the translation

( u v ) =T ( x ˜ y ˜ ) ,

then system (3.3) becomes

( x ˜ y ˜ ) ( 1 s 2 ) ( x ˜ y ˜ ) + ( f ( u , v , λ ) g ( u , v , λ ) ) ,
(3.4)

where

$$\begin{aligned}& f \bigl(u,v,\lambda^{*} \bigr) \\& \quad = \frac{a_{13}(s_{2} - a_{11}) - a_{12}a_{23}}{a_{12}(s_{2} + 1)}u^{2} + \frac{a_{14}(s_{2} - a_{11}) - a_{12}a_{24}}{a_{12}(s_{2} + 1)}uv \\& \qquad {} + \frac{a_{15}(s_{2} - a_{11}) - a_{12}a_{25}}{a_{12}(s_{2} + 1)}v^{2} + \frac{b_{1}(s_{2} - a_{11}) - a_{12}c_{1}}{a_{12}(s_{2} + 1)}u \lambda^{*} \\& \qquad {}+ \frac{b_{2}(s_{2} - a_{11}) - a_{12}c_{2}}{a_{12}(s_{2} + 1)}v\lambda^{*} + \frac{b_{3}(s_{2} - a_{11}) - a_{12}c_{3}}{a_{12}(s_{2} + 1)}u^{2} \lambda^{*} \\& \qquad {} + \frac{b_{4}(s_{2} - a_{11}) - a_{12}c_{4}}{a_{12}(s_{2} + 1)}uv\lambda^{*} + \frac{b_{5}(s_{2} - a_{11}) - a_{12}c_{5}}{a_{12}(s_{2} + 1)}v^{2} \lambda^{*} \\& \qquad {}+ \frac{e_{1}(s_{2} - a_{11}) - a_{12}d_{1}}{a_{12}(s_{2} + 1)}u^{3} + \frac{e_{2}(s_{2} - a_{11}) - a_{12}d_{2}}{a_{12}(s_{2} + 1)}u^{2}v \\& \qquad {} + \frac{e_{3}(s_{2} - a_{11}) - a_{12}d_{3}}{a_{12}(s_{2} + 1)}v^{2}u + \frac{e_{4}(s_{2} - a_{11}) - a_{12}d_{4}}{a_{12}(s_{2} + 1)}v^{3} \\& \qquad {}+ O \bigl( \bigl( \vert u \vert + \vert v \vert + \bigl\vert \lambda^{*} \bigr\vert \bigr)^{4} \bigr), \\& g \bigl(u,v,\lambda^{*} \bigr) \\& \quad = \frac{a_{13}(1 + a_{11}) + a_{12}a_{23}}{a_{12}(s_{2} + 1)}u^{2} + \frac{a_{14}(1 + a_{11}) + a_{12}a_{24}}{a_{12}(s_{2} + 1)}uv \\& \qquad {} + \frac{a_{15}(1 + a_{11}) + a_{12}a_{25}}{a_{12}(s_{2} + 1)}v^{2} + \frac{b_{1}(1 + a_{11}) + a_{12}c_{1}}{a_{12}(s_{2} + 1)}u \lambda^{*} \\& \qquad {}+ \frac{b_{2}(1 + a_{11}) + a_{12}c_{2}}{a_{12}(s_{2} + 1)}v\lambda^{*} + \frac{b_{3}(1 + a_{11}) + a_{12}c_{3}}{a_{12}(s_{2} + 1)}u^{2} \lambda^{*} \\& \qquad {} + \frac{b_{4}(1 + a_{11}) + a_{12}c_{4}}{a_{12}(s_{2} + 1)}uv\lambda^{*} + \frac{b_{5}(1 + a_{11}) + a_{12}c_{5}}{a_{12}(s_{2} + 1)}v^{2} \lambda^{*} \\& \qquad {}+ \frac{e_{1}(1 + a_{11}) + a_{12}d_{1}}{a_{12}(s_{2} + 1)}u^{3} + \frac{e_{2}(1 + a_{11}) + a_{12}d_{2}}{a_{12}(s_{2} + 1)}u^{2}v \\& \qquad {} + \frac{e_{3}(1 + a_{11}) + a_{12}d_{3}}{a_{12}(s_{2} + 1)}v^{2}u + \frac{e_{4}(1 + a_{11}) + a_{12}d_{4}}{a_{12}(s_{2} + 1)}v^{3} \\& \qquad {}+ O \bigl( \bigl( \vert u \vert + \vert v \vert + \bigl\vert \lambda^{*} \bigr\vert \bigr)^{4} \bigr), \end{aligned}$$

and

$$\begin{aligned}& u = a_{12}(\tilde{x} + \tilde{y}), \qquad v = - (1 + a_{11}) \tilde{x} + (s_{2} - a_{11})\tilde{y}, \\& uv = a_{12}[ - (1 + a_{11})\tilde{x}^{2} + (s_{2} - 2a_{11} - 1)\tilde{x}\tilde{y} + a_{12}(s_{2} - a_{11})\tilde{y}^{2}, \\& u^{2} = a_{12}^{2}(\tilde{x} + \tilde{y})^{2}, \qquad v^{2} = \bigl[ - (1 + a_{11}) \tilde{x} + (s_{2} - a_{11})\tilde{y} \bigr]^{2}, \\& uv^{2} = a_{12}(\tilde{x} + \tilde{y}) \bigl[ - (1 + a_{11})\tilde{x} + (s_{2} - a_{11})\tilde{y} \bigr]^{2}, \\& u^{3} = a_{12}^{3}(\tilde{x} + \tilde{y})^{3}, \qquad u^{2}v = a_{12}^{2}( \tilde{x} + \tilde{y})^{2} \bigl[ - (1 + a_{11})\tilde{x} + (s_{2} - a_{11})\tilde{y} \bigr], \\& v^{3} = \bigl[ - (1 + a_{11})\tilde{x} + (s_{2} - a_{11})\tilde{y} \bigr]^{3}. \end{aligned}$$

There exists a center manifold \(W_{c}(0,0,0)\) of Eq. (3.4) at the fixed point \((0, 0)\) in a small neighborhood of \(\lambda^{*}\). From the center manifold theorem, we know that there exists a center manifold

$$\begin{aligned}& W_{c}(0,0,0) \\& \quad = \bigl\{ \bigl(\tilde{x},\tilde{y},\lambda^{*} \bigr) \in R^{3},\tilde{y} = h^{*} \bigl(\tilde{x}, \lambda^{*} \bigr) = a_{1}\tilde{x}^{2} + a_{2}\tilde{x}\lambda^{*} + a_{3} \lambda^{*2} + O \bigl( \bigl( \vert \tilde{x} \vert + \bigl\vert \lambda^{*} \bigr\vert \bigr)^{3} \bigr) \bigr\} \end{aligned}$$
(3.5)

for and \(\lambda^{*}\) sufficiently small. Then the center manifold must satisfy

$$\begin{aligned}& N \bigl(h^{*} \bigl(\tilde{x},\lambda^{*} \bigr) \bigr) \\& \quad = h^{*} \bigl( - \tilde{x} + f \bigl(\tilde{x},h^{*} \bigl(\tilde{x},\lambda^{*} \bigr),\lambda^{*} \bigr), \lambda^{*} \bigr) - s_{2}h^{*} \bigl(\tilde{x}, \lambda^{*} \bigr) - g \bigl(\tilde{x},h^{*} \bigl( \tilde{x}, \lambda^{*} \bigr),\lambda^{*} \bigr) = 0. \end{aligned}$$
(3.6)

Substituting (3.4) and (3.5) into (3.6) and comparing coefficients of (3.6), we obtain where \(O ( (|\tilde{x}| + |\lambda^{*}|)^{3} )\) is a function in \((\tilde{x},\lambda^{*})\) at least of the third order, and

$$\begin{aligned}& a_{1} = - \frac{(1 + a_{11})[a_{14}(1 + a_{11}) + a_{12}a_{24}]}{1 - s_{2}^{2}} + \frac{a_{12}[(1 + a_{11})a_{13} + a_{12}a_{23}]}{1 - s_{2}^{2}} \\& \hphantom{a_{1} =}{} + \frac{[a_{15}(1 + a_{11}) + a_{12}a_{25}](1 + a_{11})^{2}}{a_{12}(1 - s_{2}^{2})}, \\& a_{2} = \frac{(1 + a_{11})[b_{2}(1 + a_{11}) + a_{12}c_{2}]}{a_{12}(1 + s_{2})^{2}} - \frac{b_{1}(1 + a_{11}) + a_{12}c_{1}}{(1 + s_{2})^{2}},\qquad a_{3} = 0. \end{aligned}$$

Therefore, model (1.2) restricted to the center manifold is given by

$$ F:\tilde{x} \to - \tilde{x} + h_{1}\tilde{x}^{2} + h_{2}\tilde{x}\lambda^{*} + h_{3} \tilde{x}^{2}\lambda^{*} + h_{4}\tilde{x} \lambda^{*2} + h_{5}\tilde{x}^{3} + O \bigl( \bigl( \vert \tilde{x} \vert + \bigl\vert \lambda^{*} \bigr\vert \bigr)^{4} \bigr), $$
(3.7)

where

$$\begin{aligned}& h_{1} = \frac{1}{a_{12}(s_{2} + 1)} \bigl\{ a_{12}^{2} \bigl[a_{13}(s_{2} - a_{11}) - a_{12}a_{23} \bigr] - a_{12}(1 + a_{11}) \bigl[a_{14}(s_{2} - a_{11}) - a_{12}a_{24} \bigr] \\& \hphantom{h_{1} =}{} + \bigl[a_{15}(s_{2} - a_{11}) - a_{12}a_{25} \bigr](1 + a_{11})^{2} \bigr\} , \\& h_{2} = \frac{1}{a_{12}(1 + s_{2})} \bigl[ \bigl[a_{12}b_{1}(s_{2} - a_{11}) - a_{12}^{2}c_{1} \bigr] - (1 + a_{11}) \bigl[b_{2}(s_{2} - a_{11}) - a_{12}c_{2} \bigr] \bigr], \\& h_{3} = \frac{a_{2}}{a_{12}(s_{2} + 1)} \bigl\{ 2a_{12}^{2} \bigl[a_{13}(s_{2} - a_{11}) - a_{12}a_{23} \bigr] \\& \hphantom{h_{3} =}{} + a_{12}(s_{2} - 1 - 2a_{11}) \bigl[a_{14}(s_{2} - a_{11}) - a_{12}a_{24} \bigr] \\& \hphantom{h_{3} =}{} - 2(1 + a_{11}) (s_{2} - a_{11}) \bigl[a_{15}(s_{2} - a_{11}) - a_{12}a_{25} \bigr] \bigr\} \\& \hphantom{h_{3} =}{}+ \frac{a_{1}}{a_{12}(s_{2} + 1)} \bigl\{ \bigl[a_{12}b_{1}(s_{2} - a_{11}) - a_{12}^{2}c_{1} \bigr] \\& \hphantom{h_{3} =}{} + (s_{2} - a_{11}) \bigl[b_{2}(s_{2} - a_{11}) - a_{12}c_{2} \bigr] \bigr\} + \frac{1}{a_{12}(s_{2} + 1)} \bigl\{ a_{12}^{2} \bigl[b_{3}(s_{2} - a_{11}) - a_{12}c_{3} \bigr] \\& \hphantom{h_{3} =}{} - a_{12}(1 + a_{11}) \bigl[b_{4}(s_{2} - a_{11}) - a_{12}c_{4} \bigr] + \bigl[b_{5}(s_{2} - a_{11}) - a_{12}c_{5} \bigr](1 + a_{11})^{2} \bigr\} , \\& h_{4} = \frac{a_{2}}{a_{12}(1 + s_{2})} \bigl\{ \bigl[a_{12}b_{1}(s_{2} - a_{11}) - a_{12}^{2}c_{1} \bigr] + (s_{2} - a_{11}) \bigl[b_{2}(s_{2} - a_{11}) - a_{12}c_{2} \bigr] \bigr\} , \\& h_{5} = \frac{1}{a_{12}(s_{2} + 1)} \bigl\{ 2a_{12}^{2}a_{1} \bigl[a_{13}(s_{2} - a_{11}) - a_{12}a_{23} \bigr] \\& \hphantom{h_{5} =}{} + a_{1}a_{12}(s_{2} - 1 - 2a_{11}) \bigl[a_{14}(s_{2} - a_{11}) - a_{12}a_{24} \bigr] \\& \hphantom{h_{5} =}{} + a_{12}^{3} \bigl[e_{1}(s_{2} - a_{11}) - a_{12}d_{1} \bigr] - a_{12}^{2}(1 + a_{11}) \bigl[e_{2}(s_{2} - a_{11}) - a_{12}d_{2} \bigr] \\& \hphantom{h_{5} =}{} - \bigl[e_{4}(s_{2} - a_{11}) - a_{12}d_{4} \bigr](1 + a_{11})^{3} \\& \hphantom{h_{5} =}{} + \bigl[e_{3}(s_{2} - a_{11}) - a_{12}d_{3} \bigr](1 + a_{11})^{2} - 2a_{1}(1 + a_{11}) (s_{2} - a_{11}) \bigl[a_{15}(s_{2} - a_{11}) - a_{12}a_{25} \bigr] \bigr\} . \end{aligned}$$

In order for map (3.7) to undergo a flip bifurcation, we require that two discriminatory quantities \(\alpha_{1}\) and \(\alpha_{2}\) are not zero, where

$$\alpha_{1} = \biggl( \frac{\partial^{2}F}{\partial \tilde{x}\partial \lambda^{*}} + \frac{1}{2} \frac{\delta F}{\delta \lambda^{*}}\frac{\delta^{2}F}{\delta \tilde{x}^{2}} \biggr)_{(0,0)} = h_{2}, $$

and

$$\alpha_{2} = \biggl( \frac{1}{6}\frac{\partial^{3}F}{\partial \tilde{x}^{3}} + \biggl( \frac{1}{2}\frac{\delta^{2}F}{\delta \tilde{x}^{2}} \biggr)^{2} \biggr)_{(0,0)} = h_{5} + h_{1}^{2}. $$

From the above analysis and the theorem of [23], we have the following result.

Theorem 3.1

If \(\alpha_{2} \ne 0\), then map (1.2) undergoes a flip bifurcation at the fixed point \((x^{ *},y^{ *} )\) when the parameter λ varies in a small neighborhood of \(\lambda_{1}\). Moreover, if \(\alpha_{2} > 0\) (resp., \(\alpha_{2} < 0\)), then the period-2 orbits that bifurcate from \((x^{ *},y^{ *} )\) are stable (resp., unstable).

In Sect. 4 we will give some values of parameters such that \(\alpha_{2} \ne 0\); thus, the flip bifurcation occurs as λ varies (see Fig. 1).

Figure 1
figure 1

(a) Bifurcation diagram of map (1.2) with \(\lambda \in ( 0, 0.55 )\), \(B = 10\), \(k = 0.95\), \(\beta = 0.2\), the initial value is \((5, 5)\); (b) Maximum Lyapunov exponents corresponding to (a).

3.2 Neimark–Sacker bifurcation analysis

Finally, we discuss the Neimark–Sacker bifurcation of \(E(x^{ *},y^{ *} )\) if parameters \(( B,k,\beta,\lambda_{2} )\) vary in a small neighborhood of HB \(H_{B}\).

Taking parameters \(( B,k,\beta,\lambda_{2} )\) arbitrarily from \(H_{B}\), we consider system (1.2) with \(( B,k,\beta,\lambda_{1} )\), which is described by

$$ \textstyle\begin{cases} x' \to x + \lambda_{2}x(B\frac{x^{\beta}}{x^{\beta} + ky^{\beta}} - x), \\ y' \to y + \lambda_{2}y(B\frac{ky^{\beta}}{x^{\beta} + ky^{\beta}} - y). \end{cases} $$
(3.8)

Map (3.8) has a unique positive fixed point \(E(x^{ *},y^{ *} )\).

Choosing \(\lambda^{*}\) as a bifurcation parameter, we consider a perturbation of (3.8) as follows:

$$ \textstyle\begin{cases} x' \to x + (\lambda_{2} + \lambda^{*})x(B\frac{x^{\beta}}{x^{\beta} + ky^{\beta}} - x), \\ y' \to y + (\lambda_{2} + \lambda^{*})y(B\frac{ky^{\beta}}{x^{\beta} + ky^{\beta}} - y), \end{cases} $$
(3.9)

where \(\vert \lambda^{*} \vert \ll 1\), which is a small perturbation parameter.

Let \(u = x - x^{*}\) and \(v = y - y^{*}\). Then we transform the fixed point \(E(x^{ *},y^{ *} )\) of map (3.9) into the origin. We have

( u v ) ( a 11 u + a 12 v + a 13 u 2 + a 14 u v + a 15 v 2 + e 1 u 3 + e 2 u 2 v + e 3 v 2 u + e 4 v 3 + O ( ( | u | + | v | ) 4 ) a 21 u + a 22 v + a 23 u 2 + a 24 u v + a 25 v 2 + d 1 u 3 + d 2 u 2 v + d 3 v 2 u + d 4 v 3 + O ( ( | u | + | v | ) 4 ) ) ,
(3.10)

where \(a_{11}\), \(a_{12}\), \(a_{13}\), \(a_{14}\), \(a_{15}\), \(e_{1}\), \(e_{2}\), \(e_{3}\), \(e_{4}\), \(a_{21}\), \(a_{22}\), \(a_{23}\), \(a_{24}\), \(a_{25}\), \(d_{1}\), \(d_{2}\), \(d_{3}\), \(d_{4}\) are given in (3.3) by substituting λ for \(\lambda_{2} + \lambda^{*}\).

Note that the characteristic equation associated with the linearization of map (3.10) at \((u,v) = (0,0)\) is given by

$$s^{2} + P \bigl(\bar{\lambda}^{*} \bigr)s + Q \bigl(\bar{ \lambda}^{*} \bigr) = 0, $$

where

$$\begin{aligned}& P \bigl(\bar{\lambda}^{*} \bigr) = - 2 - G \bigl( \lambda_{2} + \bar{\lambda}^{*} \bigr), \\& Q \bigl(\bar{\lambda}^{*} \bigr) = 1 + G \bigl(\lambda_{2} + \bar{\lambda}^{*} \bigr) + H \bigl(\lambda_{2} + \bar{ \lambda}^{*} \bigr)^{2}. \end{aligned}$$

Since parameters \(( B,k,\beta,\lambda_{2} ) \in H_{B}\), the eigenvalues of \((0, 0)\) are a pair of complex conjugate numbers s, and with modulus one by Proposition 4, where

$$s,\bar{s} = - \frac{P(\bar{\lambda}^{*})}{2} \pm \frac{i}{2}\sqrt{4Q \bigl(\bar{ \lambda}^{*} \bigr) - P^{2} \bigl(\bar{\lambda}^{*} \bigr)} = 1 + \frac{(\lambda_{2} + \lambda^{*})}{2} \pm \frac{i(\lambda_{2} + \lambda^{*})}{2}\sqrt{4H - G^{2}}. $$

Moreover, we have

$$\vert s \vert _{\bar{\lambda}^{*} = 0} = \sqrt{Q(0)} = 1,\qquad l = \frac{d \vert s \vert }{d\bar{\lambda}^{*}} |_{\bar{\lambda}^{*} = 0} = - \frac{G}{2} \ne 0. $$

Also, it requires that when \(\bar{\lambda}^{*} = 0\), \(\lambda^{m}\), \(\bar{\lambda}^{m} \ne 1\) (\(m = 1,2,3,4 \)) which is equivalent to \(P(0) \ne - 2, 0, 1, 2\). Note that \(( B,k,\beta,\lambda_{2} ) \in H_{B}\). Thus, \(P(0) \ne - 2, 2\). We only need to require that \(P(0) \ne 0, 1\), which leads to

$$ G^{2} \ne 2H, 3H. $$
(3.11)

Therefore, the eigenvalues s, of a fixed point \((0, 0)\) of (3.10) do not lie in the intersection of the unit circle with the coordinate axes when δ and (3.11) holds.

Next, we study the normal form of (3.10) at \(\bar{\lambda}^{*} = 0\).

Let \(\bar{\lambda}^{*} = 0\), \(\mu = 1 + \frac{G\lambda_{2}}{2}\), \(\omega = \frac{\lambda_{2}}{2}\sqrt{4H - G^{2}}\).

T= ( a 12 0 μ a 11 ω ) .

Moreover, using the translation

( u v ) =T ( x ˜ y ˜ ) ,

then system (3.10) becomes

( x ˜ y ˜ ) ( μ ω ω μ ) ( x ˜ y ˜ ) + ( f ˜ ( x ˜ , y ˜ ) g ˜ ( x ˜ , y ˜ ) ) ,
(3.12)

where

$$\begin{aligned}& \tilde{f}(\tilde{x},\tilde{y}) = \frac{a_{13}}{a_{12}}u^{2} + \frac{a_{14}}{a_{12}}uv + \frac{a_{15}}{a_{12}}v^{2} + \frac{e_{1}}{a_{12}}u^{3} + \frac{e_{2}}{a_{12}}u^{2}v + \frac{e_{3}}{a_{12}}v^{2}u + \frac{e_{4}}{a_{12}}v^{3} + O \bigl( \bigl( \vert \tilde{x} \vert + \vert \tilde{y} \vert \bigr)^{4} \bigr), \\& \tilde{g}(\tilde{x},\tilde{y}) = \frac{a_{13}(\mu - a_{11}) - a_{12}a_{23}}{a_{12}\omega} u^{2} + \frac{a_{14}(\mu - a_{11}) - a_{12}a_{24}}{a_{12}\omega} uv \\& \hphantom{\tilde{g}(\tilde{x},\tilde{y}) =} {} + \frac{a_{15}(\mu - a_{11}) - a_{12}a_{25}}{a_{12}\omega} v^{2} + \frac{e_{1}(\mu - a_{11}) - a_{12}d_{1}}{a_{12}\omega} u^{3} \\& \hphantom{\tilde{g}(\tilde{x},\tilde{y}) =} {}+ \frac{e_{2}(\mu - a_{11}) - a_{12}d_{2}}{a_{12}\omega} u^{2}v + \frac{e_{3}(\mu - a_{11}) - a_{12}d_{3}}{a_{12}\omega} v^{2}u \\& \hphantom{\tilde{g}(\tilde{x},\tilde{y}) =} {} + \frac{e_{4}(\mu - a_{11}) - a_{12}d_{4}}{a_{12}\omega} v^{3} + O \bigl( \bigl( \vert \tilde{x} \vert + \vert \tilde{y} \vert \bigr)^{4} \bigr), \\& u^{2} = a_{12}^{2}\tilde{x}^{2},\qquad uv = a_{12}(\mu - a_{11})\tilde{x}^{2} - a_{12}\omega \tilde{x}\tilde{y}, \\& v^{2} = (\mu - a_{11})^{2}\tilde{x}^{2} - 2(\mu - a_{11})\omega \tilde{x}\tilde{y} + \omega^{2} \tilde{y}^{2}, \\& v^{2}u = a_{12}(\mu - a_{11})^{2} \tilde{x}^{3} - 2a_{12}(\mu - a_{11})\omega \tilde{x}^{2}\tilde{y} + a_{12}\omega^{2} \tilde{x}^{2}\tilde{y}^{2}, \\& u^{2}v = a_{12}^{2}(\mu - a_{11}) \tilde{x}^{3} - a_{12}^{2}\omega \tilde{x}^{2}\tilde{y} \\& u^{3} = a_{12}^{3}\tilde{x}^{3}, \qquad v^{3} = (\mu - a_{11})^{3} \tilde{x}^{3} - 3\omega (\mu - a_{11})^{2} \tilde{x}^{2}\tilde{y} + 3\omega^{2}(\mu - a_{11}) \tilde{x}\tilde{y}^{2} - \omega^{3}\tilde{y}^{3}. \end{aligned}$$

Therefore,

$$\begin{aligned}& \tilde{f}_{\tilde{x}\tilde{x}} = 2a_{12}a_{13} + 2a_{14}(\mu - a_{11}) + \frac{2a_{15}}{a_{12}}(\mu - a_{11})^{2},\qquad \tilde{f}_{\tilde{x}\tilde{y}} = - a_{14}\omega - \frac{2a_{15}}{a_{12}}(\mu - a_{11})\omega, \\& \tilde{f}_{\tilde{y}\tilde{y}} = \frac{a_{15}}{a_{12}}\omega^{2}, \\& \tilde{f}_{\tilde{x}\tilde{x}\tilde{x}} = 6e_{1}a_{12}^{2} + 6e_{2}a_{12}(\mu - a_{11}) + 6e_{3}(\mu - a_{11})^{2} + \frac{6e_{4}}{a_{12}}(\mu - a_{11})^{3},\qquad \tilde{f}_{\tilde{y}\tilde{y}\tilde{y}} = - \frac{e_{4}}{a_{12}}\omega^{2}, \\& \tilde{f}_{\tilde{x}\tilde{y}\tilde{y}} = \frac{6}{a_{12}}\omega^{2}e_{4}( \mu - a_{11}), \\& \tilde{f}_{\tilde{x}\tilde{x}\tilde{y}} = - 4\omega e_{3}(\mu - a_{11}) - 2\omega e_{2}a_{12} - \frac{6}{a_{12}} \bigl(e_{4}\omega (\mu - a_{11})^{2} \bigr), \\& \tilde{g}_{\tilde{x}\tilde{x}} = \frac{2}{\omega} \bigl\{ a_{12} \bigl[ a_{13}(\mu - a_{11}) - a_{12}a_{23} \bigr] + (\mu - a_{11}) \bigl[ a_{14}(\mu - a_{11}) - a_{12}a_{24} \bigr] \\& \hphantom{\tilde{g}_{\tilde{x}\tilde{x}} =}{} + \bigl[ a_{15}(\mu - a_{11}) - a_{12}a_{25} \bigr](\mu - a_{11})^{2} \bigr\} , \\& \tilde{g}_{\tilde{x}\tilde{y}} = \frac{1}{a_{12}} \bigl\{ a_{12} \bigl[ a_{12}a_{24} - a_{14}(\mu - a_{11}) \bigr] - 2(\mu - a_{11}) \bigl[ a_{15}(\mu - a_{11}) - a_{12}a_{25} \bigr] \bigr\} , \\& \tilde{g}_{\tilde{y}\tilde{y}} = \frac{2\omega}{a_{12}} \bigl[ a_{15}(\mu - a_{11}) - a_{12}a_{25} \bigr], \\& \tilde{g}_{\tilde{x}\tilde{x}\tilde{x}} = \frac{6}{\omega} \bigl\{ a_{12}^{2} \bigl[ e_{1}(\mu - a_{11}) - a_{12}d_{1} \bigr] + a_{12}(\mu - a_{11}) \bigl[ e_{3}(\mu - a_{11}) - a_{12}d_{3} \bigr] \\& \hphantom{\tilde{g}_{\tilde{x}\tilde{x}\tilde{x}} =}{} + \bigl[ e_{3}(\mu - a_{11}) - a_{12}d_{3} \bigr](\mu - a_{11})^{2} \\& \hphantom{\tilde{g}_{\tilde{x}\tilde{x}\tilde{x}} =}{}+ \bigl[ e_{4}(\mu - a_{11}) - a_{12}d_{4} \bigr] (\mu - a_{11})^{3} \bigr\} , \\& \tilde{g}_{\tilde{y}\tilde{y}\tilde{y}} = \frac{6\omega^{2}}{a_{12}} \bigl[ a_{12}d_{4} - e_{4}(\mu - a_{11}) \bigr], \\& \tilde{g}_{\tilde{x}\tilde{y}\tilde{y}} = \frac{6}{a_{12}}\omega (\mu - a_{11}) \bigl[ e_{4}(\mu - a_{11}) - a_{12}d_{4} \bigr], \\& \tilde{g}_{\tilde{x}\tilde{x}\tilde{y}} = 4(\mu - a_{11}) \bigl[ a_{12}d_{3} - e_{3}(\mu - a_{11}) \bigr] + 2a_{12} \bigl[ a_{12}d_{2} - e_{2}(\mu - a_{11}) \bigr] \\& \hphantom{\tilde{g}_{\tilde{x}\tilde{x}\tilde{y}} = }{} + \frac{6}{a_{12}} \bigl[ a_{12}d_{4} - e_{4}(\mu - a_{11}) \bigr](\mu - a_{11})^{2} \end{aligned}$$

at point \((0, 0)\).

In order for system (3.12) to undergo the Neimark–Sacker bifurcation, we require that the following discriminatory quantity is not zero:

$$a = \biggl\{ - \operatorname{Re} \biggl( \frac{(1 - 2s)\bar{s}^{2}}{1 - s} \xi_{20} \xi_{11} \biggr) - \frac{1}{2} \vert \xi_{11} \vert ^{2} - \vert \xi_{02} \vert ^{2} + \operatorname{Re} (\bar{s}\xi_{21}) \biggr\} \bigg|_{\bar{\lambda}^{*} = 0}, $$

where

$$\begin{aligned}& \xi_{20} = \frac{1}{8} \bigl[ ( \tilde{f}_{\tilde{x}\tilde{x}} - \tilde{f}_{\tilde{y}\tilde{y}} + 2\tilde{g}_{\tilde{x}\tilde{y}} ) + i ( \tilde{g}_{\tilde{x}\tilde{x}} - \tilde{g}_{\tilde{y}\tilde{y}} - 2\tilde{f}_{\tilde{x}\tilde{y}} ) \bigr], \\& \xi_{11} = \frac{1}{4} \bigl[ ( \tilde{f}_{\tilde{x}\tilde{x}} + \tilde{f}_{\tilde{y}\tilde{y}} ) + i ( \tilde{g}_{\tilde{x}\tilde{x}} + \tilde{g}_{\tilde{y}\tilde{y}} ) \bigr], \\& \xi_{02} = \frac{1}{8} \bigl[ ( \tilde{f}_{\tilde{x}\tilde{x}} - \tilde{f}_{\tilde{y}\tilde{y}} - 2\tilde{g}_{\tilde{x}\tilde{y}} ) + i ( \tilde{g}_{\tilde{x}\tilde{x}} - \tilde{g}_{\tilde{y}\tilde{y}} + 2\tilde{f}_{\tilde{x}\tilde{y}} ) \bigr], \\& \xi_{21} = \frac{1}{16} \bigl[ ( \tilde{f}_{\tilde{x}\tilde{x}\tilde{x}} + \tilde{f}_{\tilde{x}\tilde{y}\tilde{y}} + \tilde{g}_{\tilde{x}\tilde{x}\tilde{y}} + \tilde{g}_{\tilde{y}\tilde{y}\tilde{y}} ) + i ( \tilde{g}_{\tilde{x}\tilde{x}\tilde{x}} + \tilde{g}_{\tilde{x}\tilde{y}\tilde{y}} - \tilde{f}_{\tilde{x}\tilde{x}\tilde{y}} - \tilde{f}_{\tilde{y}\tilde{y}\tilde{y}} ) \bigr]. \end{aligned}$$

From the above analysis, we have the following theorem.

Theorem 3.2

If condition (3.11) holds and \(a \ne 0\), then map (1.2) undergoes the Neimark–Sacker bifurcation at the fixed point \((x^{*},y^{*})\) when the parameter λ varies in a small neighborhood of \(\lambda_{2}\). Moreover, if \(a < 0\) (resp., \(a > 0\)), then an attracting (resp., repelling) invariant closed curve bifurcates from the fixed point for \(\lambda > \lambda_{2}\) (resp., \(\lambda < \lambda_{2}\)).

4 Numerical simulations

In this section, we illustrate the above analytic results and show the complex dynamical behaviors by the bifurcation diagrams, phase portraits, and maximum Lyapunov exponents for system (1.2).

For the sake of analysis, we consider the symmetric case of identical firms, let \(\lambda_{1} = \lambda_{2} = \lambda\), \(\beta_{1} = \beta_{2} = \beta\). Then, from system (1.2), we obtain

$$ \textstyle\begin{cases} x' \to x + \lambda x(B\frac{x^{\beta}}{x^{\beta} + ky^{\beta}} - x), \\ y' \to y + \lambda y(B\frac{ky^{\beta}}{x^{\beta} + ky^{\beta}} - y). \end{cases} $$
(4.1)

The bifurcation analyses are considered in the following cases:

  1. (i)

    Varying λ in range \(0 < \lambda < 0.51\) and fixing \(\beta = 0.2\), \(B = 10\), \(k = 0.95\);

  2. (ii)

    Varying B in range \(0 < B < 62.8\) and fixing \(\beta = 0.2\), \(\lambda = 0.08\), \(k = 0.95\);

  3. (iii)

    Varying β in range \(0.3 < \beta < 1\) and fixing \(k = 0.8\), \(\lambda = 0.4\), \(B = 10\);

  4. (iv)

    Varying k in range \(0.35 < k < 2.5\) and fixing \(\beta = 0.2\), \(\lambda = 0.4\), \(B = 10\).

For case (i). The bifurcation diagram of system (1.2) in the \((\lambda,x)\) plane for \(0 < \lambda < 0.51\) with initial values \((x_{0},y_{0}) = (5,5)\) is given in Fig. 1(a) to show the dynamical changes as λ varies. The maximum Lyapunov exponents corresponding to the bifurcation diagram in Fig. 1(a) are given in Fig. 1(b).

In Fig. 1, we can see that there is a stable fixed point \((5.1602,4.8398)\) for \(0 < \lambda < 0.3923\), and a flip bifurcation occurs at \(\lambda = 0.3923\). We observe that there are period-2 orbits for larger regions \(\lambda \in (0.3923,0.4571)\).

Figure 2 shows the phase portraits which are associated with Fig. 1. For \(\lambda \in ( 0, 0.55 )\), there are period-1,2,4 orbits (in Fig. 2 (a)∼(c)). Figure 2(d) shows one of the stable fixed points. Figure 2(e) shows that the Hopf bifurcations emerge from the fixed points at \(\lambda = 0.48\). When \(\lambda = 0.495\), we can see the chaotic sets in Fig. 2(f). The maximum Lyapunov exponents corresponding to \(\lambda = 0.495\) are larger than zero, which confirms the existence of the chaotic sets in Fig. 1(b).

Figure 2
figure 2

Phase plane for map (1.2) with \(B = 10\), \(k = 0.95\), \(\beta = 0.2\) and different λ.

For case (ii). The bifurcation diagram of system (1.2) in the \((B,x)\) plane for \(0 < B < 62.8\) with initial values \((x_{0},y_{0}) = (5,5)\) is given in Fig. 3(a) to show the dynamical changes as B varies. The maximum Lyapunov exponents corresponding to the bifurcation diagram in Fig. 3(a) are given in Fig. 3(b).

Figure 3
figure 3

(a) Bifurcation diagram of map (1.2) with \(B \in ( 0, 62.8 )\), \(\beta = 0.2\), \(k = 0.95\), \(\lambda = 0.08\), the initial value is \((5, 5)\); (b) Maximum Lyapunov exponents corresponding to (a).

In Fig. 3, a flip bifurcation occurs at \(B = 49.38\) by Proposition 1. We observe that there are period-2 orbits for larger regions \(B \in (49.38,57.35)\). Other cases are similar to case (i).

For case (iii). The bifurcation diagram of system (1.2) in the \((\beta,x)\) plane for \(0 < \beta < 1\) with initial values \((x_{0},y_{0}) = (5,5)\) is given in Fig. 3(a) to show the dynamical changes as β varies. The maximum Lyapunov exponents corresponding to the bifurcation diagram in Fig. 4(a) are given in Fig. 4(b).

Figure 4
figure 4

(a) Bifurcation diagram of map (1.2) with \(\beta \in ( 0, 1 )\), \(k = 0.8\), \(B = 10\), \(\lambda = 0.35\), the initial value is \((5, 5)\); (b) Maximum Lyapunov exponents corresponding to (a).

In Fig. 4, we can see that a flip bifurcation occurs at \(\beta = 0.6589\) by Proposition 1. We observe that there are period-2 orbits for larger regions \(\beta \in (0.6589,0.903)\). Other cases are similar to case (i).

For case (iv). The bifurcation diagram of system (1.2) in the \((k,x)\) plane for \(0.355 < k < 2.5\) with initial values \((x_{0},y_{0}) = (5,5)\) is given in Fig. 5(a) to show the dynamical changes as k varies. The maximum Lyapunov exponents corresponding to the bifurcation diagram in Fig. 5(a) are given in Fig. 5(b).

Figure 5
figure 5

(a) Bifurcation diagram of map (1.2) with \(k \in ( 0.35, 2.5 )\), \(\beta = 0.2\), \(B = 10\), \(\lambda = 0.4\), the initial value is \((5, 5)\); (b) Maximum Lyapunov exponents corresponding to (a).

In Fig. 5, there exist double period doubling bifurcations and chaos. We can see that a flip bifurcation occurs at \(k = 0.9773\) or \(k = 1.027\) by Proposition 1. We observe that there are period-2 orbits for regions \(k \in (0.6261,0.9773)\) or \(k \in (1.027,1.613)\). There are period-1, 2, 4, 6, 8, 16 orbits with \(k = 1,0.7,0.58, 1.95,0.56,0.55\). When \(k = 0.45,1.9,2.15\), we can see the chaotic sets in Fig. 5(a). The maximum Lyapunov exponents corresponding to them are greater than zero, which implies the existence of the chaotic sets in Fig. 5(b).

5 Discussion

In this paper, we discuss the dynamical behaviors of model (1.2). From the discussion in Sect. 2, we know that there exist flip bifurcation and chaos about equilibrium as the parameters vary in the small neighborhood. We have obtained a global qualitative analysis of model (1.2) depending on all parameters and showed that the model exhibits the bifurcations. By choosing λ, B, β, k as bifurcation parameters, respectively, it was shown that the model undergoes a series of bifurcations including the flip bifurcation, period doubling bifurcation, and chaos. Moreover, system (1.2) exhibits many complex dynamic behaviors, including period-1, 2, 4, 6, 8, 16 orbits, invariant cycle, a cascade of period-doubling, quasi-periodic orbits, and the chaotic sets. These results reveal far richer dynamics of the discrete model compared to the continuous model.