Evolutionary Dynamics of Cooperation in the Public Goods Game with Individual Disguise and Peer Punishment

Abstract

The phenomenon of individual disguise is ubiquitous in the real world. Recent evidences show that peer punishment is successful to stabilize cooperation among selfish individuals. However, it is unclear whether peer punishment is still able to stabilize cooperation when individual disguise for escaping being punished is considered. In this paper, we thus introduce individual disguise of defectors into the public goods game with peer punishment and aim to explore how peer punishment influences the evolutionary dynamics of cooperation, defection, and punishment in this scenario. We consider both cases of infinite and finite populations. In infinite populations, by using replicator equations, we find that although low disguise cost can hinder public cooperation, peer punishment still plays a positive role in promoting the evolution of cooperation, no matter whether second-order punishment is considered or not. We further demonstrate that the larger fine on defectors or the smaller the cost of punishment, the easier to establish a state of full punishment. In addition, in finite populations we reveal that the population spends most of the time in the region of defection for low disguise cost, and the population spends most of the time in the region of cooperation for high disguise. When second-order punishment is not considered, the punishment strategy does not have evolutionary advantage, whereas when second-order punishment is considered, the population can evolve toward regime of punishment and spends most of the time in the monomorphic states with widespread punishment or cooperation.

Appendices

Appendix A Evolutionary Dynamics in Infinite Populations Without Second-Order Punishment

To study the replicator dynamics in infinite populations without second-order punishment, we first compute the average payoffs of the three strategies by substituting the payoff equations (1) of three strategies into Eq. (3), which are, respectively, given as

$$\begin{aligned} P_{C}&=\frac{rc(N-1)(x+z)}{N}-c+\frac{rc}{N},\nonumber \\ P_{D}&=\frac{rc(N-1)(x+z)}{N}-q\gamma -(1-pq)\alpha (N-1)z,\nonumber \\ P_{P}&=\frac{rc(N-1)(x+z)}{N}-c+\frac{rc}{N}-(1-pq)\beta (N-1)y. \end{aligned}$$

(15)

Accordingly, by substituting $x+y+z=1$ and $\bar{P}=xP_{C}+yP_{D}+zP_{P}$ into Eq. (4), we have the following equation system

$$\begin{aligned} \Big \{\begin{array}{c} \dot{x}=x[(1-x)(P_{C}-P_{P})-y(P_{D}-P_{P})] \\ \dot{y}=y[(1-y)(P_{D}-P_{P})-x(P_{C}-P_{P})], \end{array} \end{aligned}$$

(16)

where $P_{C}-P_{P}=(1-pq)\beta (N-1)y$ and $P_{D}-P_{P}=c-\frac{rc}{N}+(1-pq)\beta (N-1)y-q\gamma -(1-pq)\alpha (N-1)(1-x-y)$.

From the equation (16), we can know that (0, 1), (1, 0), (0, 0), $(x_{0},0)$, and $(0,\frac{q\gamma -\sigma +(1-pq)\alpha (N-1)}{(1-pq)(\alpha +\beta )(N-1)})$ are boundary equilibrium points of the system for $\sigma \ne q\gamma$ and $pq\ne 1$, where $\sigma =c-\frac{rc}{N}$ and $0<x_{0}<1$. In addition, there are no interior equilibrium points for the equation system (16). Furthermore, we study the stability of these equilibria by judging the sign of the eigenvalues of Jacobian matrix [34], and the Jacobian matrix of equation system (16) can be written by calculating the first-order partial derivatives as

$$\begin{aligned} J=\left( \begin{array}{cc} \frac{\partial H(x,y)}{\partial x} &{} \frac{\partial H(x,y)}{\partial y} \\ \frac{\partial K(x,y)}{\partial x} &{} \frac{\partial K(x,y)}{\partial y} \\ \end{array} \right) , \end{aligned}$$

where $\dot{x}=H(x,y)$ and $\dot{y}=K(x,y)$. Correspondingly, we have

$$\begin{aligned} \frac{\partial H(x,y)}{\partial x}&=(1-x)(1-pq)\beta (N-1)y-y[\sigma +(1-pq)\beta (N-1)y-q\gamma \\&\quad -(1-pq)\alpha (N-1)(1-x-y)]-(1-pq)(\beta +\alpha )(N-1)xy,\\ \frac{\partial H(x,y)}{\partial y}&=x[(1-x)(1-pq)\beta (N-1)-\sigma -(1-pq)\beta (N-1)y+q\gamma \\&\quad +(1-pq)\alpha (N-1)(1-x-y)-(1-pq)(\beta +\alpha )(N-1)y],\\ \frac{\partial K(x,y)}{\partial x}&=y[(1-y)(1-pq)\alpha (N-1)-(1-pq)\beta (N-1)y], \end{aligned}$$

and

$$\begin{aligned} \frac{\partial K(x,y)}{\partial y}&=(1-y)[\sigma +(1-pq)\beta (N-1)y-q\gamma -(1-pq)\alpha (N-1)\\&\quad (1-x-y)]-(1-pq)\beta (N-1)xy+y[-\sigma -(1-pq)\beta (N-1)y\\&\quad +q\gamma -(1-pq)\alpha (N-1)x+2(1-y)(1-pq)\alpha (N-1)\\&\quad +(1-y)(1-pq)\beta (N-1)-(1-pq)\beta (N-1)x]. \end{aligned}$$

Thus the specific forms of the Jacobian matrix at the above four boundary equilibrium points are, respectively,

$$\begin{aligned} J(0,0)= & {} \left( \begin{array}{cc} 0 &{} 0 \\ 0 &{} \sigma -q\gamma -(1-pq)\alpha (N-1) \\ \end{array} \right) , \end{aligned}$$

(17)

$$\begin{aligned} J(1,0)= & {} \left( \begin{array}{cc} 0 &{} -\sigma +q\gamma \\ 0 &{} \sigma -q\gamma \\ \end{array} \right) , \end{aligned}$$

(18)

$$\begin{aligned} J(0,1)= & {} \left( \begin{array}{cc} -\sigma +q\gamma &{} 0 \\ -(1-pq)\beta (N-1) &{} -\sigma +q\gamma -(1-pq)\beta (N-1) \\ \end{array} \right) , \end{aligned}$$

(19)

$$\begin{aligned} J(x_{0},0)= & {} \left( \begin{array}{cc} 0 &{} x_{0}[-\sigma +q\gamma +(1-x_{0})(1-pq)(N-1)(\alpha +\beta )] \\ 0 &{} \sigma -q\gamma -(1-pq)\alpha (N-1)(1-x_{0}) \\ \end{array} \right) , \end{aligned}$$

(20)

and

$$\begin{aligned} J(0,\frac{q\gamma +(1-pq)\alpha (N-1)-\sigma }{(1-pq)(\alpha +\beta )(N-1)})=\left( \begin{array}{cc} a &{} 0 \\ b &{} d \\ \end{array} \right) , \end{aligned}$$

(21)

where $a=\frac{\beta q\gamma +(1-pq)\beta \alpha (N-1)-\sigma \beta }{\beta +\alpha }$, $b=\frac{q\gamma +(1-pq)\alpha (N-1)-\sigma }{(1-pq)(\beta +\alpha )(N-1)}(\sigma -q\gamma )$, and $d=\frac{q\gamma +(1-pq)\alpha (N-1)-\sigma }{(1-pq)(\beta +\alpha )(N-1)}[(1-pq)\beta (N-1)-q\gamma +\sigma ]$.

We accordingly have the following results.

• (1) In the condition of $-(1-pq)\alpha (N-1)<q\gamma -\sigma <0$, (1, 0), $(0,\frac{q\gamma -\sigma +(1-pq)\alpha (N-1)}{(1-pq)(\alpha +\beta )(N-1)})$, (0, 1), $(x_{0},0)$, and (0, 0) are equilibrium points. Since the largest eigenvalues of J(1, 0) and $J(0,\frac{q\gamma -\sigma +(1-pq)\alpha (N-1)}{(1-pq)(\alpha +\beta )(N-1)})$ are both positive, the equilibrium points (1, 0) and $(0,\frac{q\gamma -\sigma +(1-pq)\alpha (N-1)}{(1-pq)(\alpha +\beta )(N-1)})$ are unstable, whereas the largest eigenvalue of J(0, 1) is negative, so the equilibrium point (0, 1) is stable. In addition, on the basis of Refs. [4, 26, 45], we can know that the PC edge consists of equilibrium points $(x_{0},0)$, and all those with $\frac{\sigma -q\gamma }{(1-pq)\alpha (N-1)}<z\le 1$ are stable, but not asymptotically stable. Correspondingly, the equilibrium point (0, 0) is stable on the edge of PC for $-(1-pq)\alpha (N-1)<q\gamma -\sigma <0$.

• (2) In the condition of $0<q\gamma -\sigma <(1-pq)\beta (N-1)$, (1, 0), $(0,\frac{q\gamma -\sigma +(1-pq)\alpha (N-1)}{(1-pq)(\alpha +\beta )(N-1)})$, (0, 1), $(x_{0},0)$, and (0, 0) are equilibrium points. Since the largest eigenvalues of J(0, 1) and $J(0,\frac{q\gamma -\sigma +(1-pq)\alpha (N-1)}{(1-pq)(\alpha +\beta )(N-1)})$ are both positive, the equilibrium points (0, 1) and $(0,\frac{q\gamma -\sigma +(1-pq)\alpha (N-1)}{(1-pq)(\alpha +\beta )(N-1)})$ are unstable. Similarly, on the basis of Refs. [4, 26, 45], we can know that the equilibrium points (0, 0) and $(x_{0},0)$ are stable on the edge of PC. Furthermore, since $P_{D}-P_{C}=-q\gamma +\sigma -(1-pq)\alpha (N-1)z<0$ in the interior phase space, we have $\dot{x}=x(1-x)(P_{C}-P_{D})>0$ and $\dot{y}=y(1-y)(P_{D}-P_{C})<0$ according to Eq. (16), which reveal that x will always evolve into the state $x=1$ for small perturbations. Correspondingly, the equilibrium point (1, 0) is stable for $0<q\gamma -\sigma <(1-pq)\beta (N-1)$.

• (3) In the condition of $q\gamma -\sigma \ge (1-pq)\beta (N-1)$, (1, 0), (0, 1), $(x_{0},0)$, and (0, 0) are equilibrium points. Since the largest eigenvalue of J(0, 1) is positive, the equilibrium point (0, 1) is unstable. In addition, similar to the case of $0<q\gamma -\sigma <(1-pq)\beta (N-1)$, we can know that the equilibrium points (0, 0) and $(x_{0},0)$ are stable on the edge of PC, and the equilibrium point (1, 0) is also stable.

• (4) In the condition of $q\gamma -\sigma <-(1-pq)\alpha (N-1)$, (1, 0), (0, 1), $(x_{0},0)$, and (0, 0) are equilibrium points. Since the largest eigenvalues of J(0, 0), $J(x_{0},0)$, and J(1, 0) are all positive, the equilibrium points (0, 0), $(x_{0},0)$, and (1, 0) are unstable, whereas the largest eigenvalue of J(0, 1) is negative, so the equilibrium point (0, 1) is stable for $q\gamma -\sigma <-(1-pq)\alpha (N-1)$.

• (5) In the condition of $q\gamma -\sigma =-(1-pq)\alpha (N-1)$, (1, 0), (0, 1), $(x_{0},0)$, and (0, 0) are equilibrium points. Since the largest eigenvalues of J(1, 0) and $J(x_{0},0)$ are positive, the equilibrium points (1, 0) and $(x_{0},0)$ are unstable, whereas the largest eigenvalue of J(0, 1) is negative, so the equilibrium point (0, 1) is stable. In addition, since $P_{P}-P_{C}=-(1-pq)\beta (N-1)y<0$ and $P_{D}-P_{C}=(1-pq)\alpha (N-1)(1-z)>0$ in the interior phase space, the equilibrium point (0, 0) is unstable for $q\gamma -\sigma =-(1-pq)\alpha (N-1)$.

In addition to the above cases, we consider the special case of $\sigma =q\gamma$. From the equation (16), we can know that (0, 1), (1, 0), (0, 0), $(x_{0},0)$, and $(0,\frac{(1-pq)\alpha (N-1)}{(1-pq)(\alpha +\beta )(N-1)})$ are boundary equilibrium points of the system for $pq\ne 1$. Since the largest eigenvalue of $J(0,\frac{(1-pq)\alpha (N-1)}{(1-pq)(\alpha +\beta )(N-1)})$ is positive, the equilibrium point $(0,\frac{(1-pq)\alpha (N-1)}{(1-pq)(\alpha +\beta )(N-1)})$ is unstable. In addition, we can know that there exist a continuum of equilibria on the edges of PC and CD. Similarly, on the basis of Refs. [4, 26, 45], the equilibria of PC edge are stable, but not asymptotically stable. Correspondingly, the equilibrium point (0, 0) is stable on the edges of PC. Furthermore, since $P_{D}<P_{C}$ in the interior phase space, then $\frac{d(y/x)}{dt}=(y/x)(P_{D}-P_{C})<0$. Accordingly, the frequency ratio of the defectors to cooperators decreases over time. Therefore, if rare punishers are introduced, the system into a stable population of the two types of defectors and cooperators. In other words, the equilibria of CD edge are stable, but not asymptotically stable. Besides, since $P_{P}<P_{C}$ in the interior phase space, the equilibrium point (1, 0) can be the eventually evolutionary outcome.

Appendix B Evolutionary Dynamics in Infinite Populations with Second-Order Punishment

To study the replicator dynamics in infinite populations with second-order punishment, we first compute the average payoffs of the three strategies by substituting the payoff equations (2) of three strategies into Eq. (3), which are, respectively, given as

$$\begin{aligned} P_{C}&=\frac{rc(N-1)(x+z)}{N}-c+\frac{rc}{N}-\alpha (N-1)z,\nonumber \\ P_{D}&=\frac{rc(N-1)(x+z)}{N}-q\gamma -(1-pq)\alpha (N-1)z,\nonumber \\ P_{P}&=\frac{rc(N-1)(x+z)}{N}-c+\frac{rc}{N}-(1-pq)\beta (N-1)y-\beta (N-1)x. \end{aligned}$$

(22)

Correspondingly, by substituting $x+y+z=1$ and $\bar{P}=xP_{C}+yP_{D}+zP_{P}$ into Eq. (4), we have the following equation system

$$\begin{aligned} \Big \{\begin{array}{c} \dot{x}=x[(1-x)(P_{C}-P_{P})-y(P_{D}-P_{P})] \\ \dot{y}=y[(1-y)(P_{D}-P_{P})-x(P_{C}-P_{P})], \end{array} \end{aligned}$$

(23)

where $P_{C}-P_{P}=(1-pq)\beta (N-1)y+\beta (N-1)x-\alpha (N-1)(1-x-y)$ and $P_{D}-P_{P}=c-\frac{rc}{N}+(1-pq)\beta (N-1)y+\beta (N-1)x-q\gamma -(1-pq)\alpha (N-1)(1-x-y)$.

The equation system (23) has at most six equilibrium points which are (0, 0), (0, 1), (1, 0), $(0,\frac{q\gamma -\sigma +(1-pq)\alpha (N-1)}{(1-pq)(\alpha +\beta )(N-1)})$, $(\frac{\alpha }{\alpha +\beta },0)$, and $(x^{*},y^{*})$ for $\sigma \ne q\gamma$, $pq\ne 0$, and $pq\ne 1$, respectively, where $x^{*}=1-\frac{q\gamma -\sigma }{\alpha pq(N-1)}-\frac{1}{pq}+\frac{(q\gamma -\sigma )(\alpha +\beta )}{\alpha \beta (pq)^{2}(N-1)}$ and $y^{*}=\frac{1}{pq}-\frac{(q\gamma -\sigma )(\alpha +\beta )}{\alpha \beta (pq)^{2}(N-1)}$. Here the first five are boundary equilibrium points, while the last one is an interior equilibrium point. In addition, the stability of six equilibria can be determined by the sign of eigenvalues of Jacobian matrix [34], and the Jacobian matrix of the equation system (23) can be obtained as

$$\begin{aligned} J=\left( \begin{array}{cc} \frac{\partial W(x,y)}{\partial x} &{} \frac{\partial W(x,y)}{\partial y} \\ \frac{\partial N(x,y)}{\partial x} &{} \frac{\partial N(x,y)}{\partial y} \\ \end{array} \right) , \end{aligned}$$

where $\dot{x}=W(x,y)$ and $\dot{y}=N(x,y)$. Correspondingly, we have

$$\begin{aligned} \frac{\partial W(x,y)}{\partial x}&=(1-x)[(1-pq)\beta (N-1)y+\beta (N-1)x-\alpha (N-1)(1-x-y)]\\&\quad -y[\sigma +(1-pq)\beta (N-1)y+\beta (N-1)x-q\gamma -(1-pq)\alpha (N-1)\\&\quad (1-x-y)]+x\{-(1-pq)\beta (N-1)y-\beta (N-1)x+\alpha (N-1)\\&\quad (1-x-y)+(1-x)(N-1)(\beta +\alpha )-y(N-1)[\beta +(1-pq)\alpha ]\},\\ \frac{\partial W(x,y)}{\partial y}&=x\{-\sigma -(1-pq)\beta (N-1)y-\beta (N-1)x+q\gamma +(1-pq)\alpha (N-1)\\&\quad (1-x-y)-y[(1-pq)\beta (N-1)+(1-pq)\alpha (N-1)]+(1-x)[\\&\quad (1-pq)\beta (N-1)+\alpha (N-1)]\},\\ \frac{\partial N(x,y)}{\partial x}&=y\{(1-y)(N-1)[\beta +(1-pq)\alpha ]-(1-pq)\beta (N-1)y\\&\quad -\beta (N-1)x+\alpha (N-1)(1-x-y)-x(N-1)(\beta +\alpha )\}, \end{aligned}$$

and

$$\begin{aligned} \frac{\partial N(x,y)}{\partial y}&=(1-y)[\sigma +(1-pq)\beta (N-1)y+\beta (N-1)x-q\gamma -(1-pq)\alpha \\&\quad (N-1)(1-x-y)]-x(N-1)[(1-pq)\beta y+\beta x-\alpha (1-x-y)]\\&\quad +y\{-\sigma -(1-pq)\beta (N-1)y-\beta (N-1)x+q\gamma +(1-pq)\alpha (N-1)\\&\quad (1-x-y)+(1-y)(1-pq)(N-1)(\beta +\alpha )-x(N-1)[(1-pq)\beta \\&\quad +\alpha ]\}. \end{aligned}$$

Thus the specific forms of the Jacobian matrix at the above six equilibrium points are, respectively,

$$\begin{aligned}&J(0,0)=\left( \begin{array}{cc} -\alpha (N-1) &{} 0 \\ 0 &{} \sigma -q\gamma -(1-pq)\alpha (N-1) \\ \end{array} \right) , \end{aligned}$$

(24)

$$\begin{aligned}&J(1,0)=\left( \begin{array}{cc} -\beta (N-1) &{} -\sigma +q\gamma -\beta (N-1) \\ 0 &{} \sigma -q\gamma \\ \end{array} \right) , \end{aligned}$$

(25)

$$\begin{aligned}&J(0,1)=\left( \begin{array}{cc} -\sigma +q\gamma &{} 0 \\ -(1-pq)\beta (N-1) &{} -\sigma +q\gamma -(1-pq)\beta (N-1) \\ \end{array} \right) , \end{aligned}$$

(26)

$$\begin{aligned}&J(0,\frac{q\gamma +(1-pq)\alpha (N-1)-\sigma }{(1-pq)(\alpha +\beta )(N-1)})=\left( \begin{array}{cc} a &{} 0 \\ b &{} d \\ \end{array} \right) , \end{aligned}$$

(27)

$$\begin{aligned}&J(\frac{\alpha }{\alpha +\beta },0)=\left( \begin{array}{cc} \frac{\alpha \beta (N-1)}{\alpha +\beta } &{} a_{1} \\ 0 &{} b_{1} \\ \end{array} \right) , \end{aligned}$$

(28)

and

$$\begin{aligned} J(x^{*},y^{*})=\left( \begin{array}{cc} a_{2} &{} b_{2} \\ c_{2} &{} d_{2} \\ \end{array} \right) , \end{aligned}$$

(29)

where $a=\frac{[(1-pq)\beta +\alpha ](q\gamma -\sigma )-pq(1-pq)\alpha \beta (N-1)}{(1-pq)(\alpha +\beta )}$,

$b=\frac{[q\gamma -\sigma +(1-pq)\alpha (N-1)][(\alpha +\beta )(-q\gamma +\sigma )(2-pq)+(1-pq)(N-1)(\beta ^{2}+\alpha \beta )]}{[(1-pq)(\alpha +\beta )]^{2}(N-1)}$,

$d=\frac{[q\gamma +(1-pq)\alpha (N-1)-\sigma ][(1-pq)\beta (N-1)-q\gamma +\sigma ]}{(1-pq)(\alpha +\beta )(N-1)}$, $a_{1}=[q\gamma -\sigma -\frac{pq\alpha \beta (N-1)}{\alpha +\beta }+ \frac{\beta (N-1)[(1-pq)\beta +\alpha ]}{\alpha +\beta }]\frac{\alpha }{\alpha +\beta }$, $b_{1}=\sigma -q\gamma +\frac{\alpha \beta (N-1)}{\alpha +\beta }-\frac{(1-pq)\alpha \beta (N-1)}{\alpha +\beta }$, $a_{2}=x^{*}\{(1-x^{*})(N-1)(\alpha +\beta )-y^{*}(N-1)[\beta +(1-pq)\alpha ]\}$, $b_{2}=x^{*}\{-y^{*}(1-pq)(N-1)(\alpha +\beta )+(1-x^{*})(N-1)[(1-pq)\beta +\alpha ]\}$, $c_{2}=y^{*}\{(1-y^{*})(N-1)[\beta +(1-pq)\alpha ]-x^{*}(N-1)(\beta +\alpha )\}$, and $d_{2}=y^{*}\{(1-y^{*})(1-pq)(N-1)(\alpha +\beta )-x^{*}(N-1)[(1-pq)\beta +\alpha ]\}$.

We accordingly have the following results.

• (1) In the condition of $\frac{pq\alpha \beta (N-1)(1-pq)}{\alpha +\beta -pq\beta }<q\gamma -\sigma <\min \{(1-pq)\beta (N-1),\frac{pq\alpha \beta (N-1)}{\alpha +\beta }\}$, the equation system (23) has six equilibrium points which are (0, 0), (0, 1), (1, 0), $(0,\frac{q\gamma +(1-pq)\alpha (N-1)-\sigma }{(1-pq)(\alpha +\beta )(N-1)})$, $(\frac{\alpha }{\alpha +\beta },0)$, and $(x^{*},y^{*})$, respectively. Since the largest eigenvalues of $J(0,\frac{q\gamma -\sigma +(1-pq)\alpha (N-1)}{(1-pq)(\alpha +\beta )(N-1)})$, $(\frac{\alpha }{\alpha +\beta },0)$, and J(0, 1) are positive, the equilibrium points $(\frac{\alpha }{\alpha +\beta },0)$, (0, 1), and $(0,\frac{q\gamma -\sigma +(1-pq)\alpha (N-1)}{(1-pq)(\alpha +\beta )(N-1)})$ are unstable, whereas the largest eigenvalues of J(0, 0) and J(1, 0) are both negative, so the equilibrium points (0, 0) and (1, 0) are stable. In addition, for the equilibrium point $(x^{*},y^{*})$, we assume that $\lambda _{1}$ and $\lambda _{2}$ are two eigenvalues of $J(x^{*},y^{*})$. We thus have $\lambda _{1}+\lambda _{2}=\frac{\partial W(x,y)}{\partial x}|_{(x^{*},y^{*})}+\frac{\partial N(x,y)}{\partial y}|_{(x^{*},y^{*})}=(\alpha +\beta )(N-1)(1-x^{*}-y^{*})(x^{*}+y^{*}-y^{*}pq)>0$, and correspondingly the equilibrium point $(x^{*},y^{*})$ is unstable.

• (2) In the condition of $(1-pq)\beta (N-1)\le q\gamma -\sigma <\frac{pq\alpha \beta (N-1)}{\alpha +\beta }$, the equation system (23) has five equilibrium points which are (0, 0), (0, 1), (1, 0), $(\frac{\alpha }{\alpha +\beta },0)$, and $(x^{*},y^{*})$, respectively. Similar to the case of $\frac{pq\alpha \beta (N-1)(1-pq)}{\alpha +\beta -pq\beta }<q\gamma -\sigma <\min \{(1-pq)\beta (N-1),\frac{pq\alpha \beta (N-1)}{\alpha +\beta }\}$, we can know that the equilibrium points $(\frac{\alpha }{\alpha +\beta },0)$, (0, 1), and $(x^{*},y^{*})$ are unstable, while the equilibrium points (0, 0) and (1, 0) are stable for $(1-pq)\beta (N-1)\le q\gamma -\sigma <\frac{pq\alpha \beta (N-1)}{\alpha +\beta }$.

• (3) In the condition of $-(1-pq)\alpha (N-1)<q\gamma -\sigma <0$, the equation system (23) has five equilibrium points which are (0, 0), (0, 1), (1, 0), $(0,\frac{q\gamma +(1-pq)\alpha (N-1)-\sigma }{(1-pq)(\alpha +\beta )(N-1)})$, and $(\frac{\alpha }{\alpha +\beta },0)$, respectively. Since the largest eigenvalues of J(1, 0), $J(\frac{\alpha }{\alpha +\beta },0)$, and $J(0,\frac{q\gamma -\sigma +(1-pq)\alpha (N-1)}{(1-pq)(\alpha +\beta )(N-1)})$ are positive, the equilibrium points $(\frac{\alpha }{\alpha +\beta },0)$, (1, 0), and $(0,\frac{q\gamma -\sigma +(1-pq)\alpha (N-1)}{(1-pq)(\alpha +\beta )(N-1)})$ are unstable, whereas the two eigenvalues of J(0, 1) and J(0, 0) are both negative; consequently, the equilibrium points (0, 1) and (0, 0) are stable.

• (4) In the conditions of $0<q\gamma -\sigma \le \frac{pq\alpha \beta (N-1)(1-pq)}{\alpha +\beta -pq\beta }$ or $\frac{pq\alpha \beta (N-1)}{\alpha +\beta }\le q\gamma -\sigma <(1-pq)\beta (N-1)$, the equation system (23) has five equilibrium points which are (0, 0), (0, 1), (1, 0), $(0,\frac{q\gamma +(1-pq)\alpha (N-1)-\sigma }{(1-pq)(\alpha +\beta )(N-1)})$, and $(\frac{\alpha }{\alpha +\beta },0)$, respectively. Since $\sigma <q\gamma$, the largest eigenvalues of J(0, 1), $J(\frac{\alpha }{\alpha +\beta },0)$, and $J(0,\frac{q\gamma -\sigma +(1-pq)\alpha (N-1)}{(1-pq)(\alpha +\beta )(N-1)})$ are positive, the equilibrium points $(\frac{\alpha }{\alpha +\beta },0)$, (0, 1), and $(0,\frac{q\gamma -\sigma +(1-pq)\alpha (N-1)}{(1-pq)(\alpha +\beta )(N-1)})$ are unstable, whereas the two eigenvalues of J(1, 0) and J(0, 0) are both negative; consequently, the equilibrium points (1, 0) and (0, 0) are stable.

• (5) In the condition of $q\gamma -\sigma \le -(1-pq)\alpha (N-1)$, the equation system (23) has four equilibrium points which are (0, 0), (0, 1), (1, 0), and $(\frac{\alpha }{\alpha +\beta },0)$, respectively. For $q\gamma -\sigma +(1-pq)\alpha (N-1)<0$, the largest eigenvalues of J(1, 0), J(0, 0), and $J(\frac{\alpha }{\alpha +\beta },0)$ are positive, so the equilibrium points (1, 0), (0, 0), and $(\frac{\alpha }{\alpha +\beta },0)$ are unstable, whereas the largest eigenvalue of J(0, 1) is negative, so the equilibrium point (0, 1) is stable. For $q\gamma -\sigma +(1-pq)\alpha (N-1)=0$, since the largest eigenvalues of J(1, 0) and $J(\frac{\alpha }{\alpha +\beta },0)$ are positive, the equilibrium points (1, 0) and $(\frac{\alpha }{\alpha +\beta },0)$ are unstable. Whereas the largest eigenvalue of J(0, 1) is negative, so the equilibrium point (0, 1) is stable. In addition, we cannot determine the stability of the equilibrium point (0, 0) based only on the sign of the eigenvalues of J(0, 0). Instead, we study its stability by using the center manifold theorem [34]. To do that, the equation system (23) becomes

  $$\begin{aligned} \dot{x}&=x\{(1-x)[(1-pq)\beta (N-1)y+\beta (N-1)x-\alpha (N-1)(1-x-y)]\nonumber \\&\quad -y[(1-pq)\alpha (N-1)(x+y)+(1-pq)\beta (N-1)y+\beta (N-1)x]\},\nonumber \\ \dot{y}&=y\{(1-y)[(1-pq)\alpha (N-1)(x+y)+(1-pq)\beta (N-1)y+\beta (N-1)x]\nonumber \\&\quad -x[(1-pq)\beta (N-1)y+\beta (N-1)x-\alpha (N-1)(1-x-y)]\}. \end{aligned}$$

  (30)

  Using the center manifold theorem [34], we have that $x=h(y)$ is the center manifold for the system (30). We start to try $h(y)=O(|y|^{2})$, which results in the reduced system as

  $$\begin{aligned} \dot{y}=(y^{2}-y^{3})(1-pq)(N-1)(\alpha +\beta )+O(|y|^{4}). \end{aligned}$$

  (31)

  Since $(1-pq)(N-1)(\alpha +\beta )\ne 0$, the equilibrium point (0, 0) is unstable.

• (6) In the condition of $q\gamma -\sigma \ge \max \{(1-pq)\beta (N-1),\frac{pq\alpha \beta (N-1)}{\alpha +\beta }\}$, the equation system (23) has four equilibrium points which are (0, 0), (0, 1), (1, 0), and $(\frac{\alpha }{\alpha +\beta },0)$, respectively. Since the largest eigenvalues of J(0, 1) and $J(\frac{\alpha }{\alpha +\beta },0)$ are both positive, the equilibrium points (0, 1) and $(\frac{\alpha }{\alpha +\beta },0)$ are unstable, whereas the largest eigenvalues of J(0, 0) and J(1, 0) are both negative; consequently, the equilibrium points (0, 0) and (1, 0) are stable.

Finally, for the special case of $\sigma =q\gamma$, from Eq. (23), we can know that (0, 0), (0, 1), (1, 0), $(0,\frac{(1-pq)\alpha (N-1)}{(1-pq)(\alpha +\beta )(N-1)})$, and $(\frac{\alpha }{\alpha +\beta },0)$ are boundary equilibrium points of the system for $pq\ne 1$. Since the largest eigenvalues of $J(\frac{\alpha }{\alpha +\beta },0)$ and $J(0,\frac{(1-pq)\alpha (N-1)}{(1-pq)(\alpha +\beta )(N-1)})$ are both positive, the largest eigenvalue of J(0, 0) is negative. Consequently, the equilibrium points $(\frac{\alpha }{\alpha +\beta },0)$ and $(0,\frac{(1-pq)\alpha (N-1)}{(1-pq)(\alpha +\beta )(N-1)})$ are unstable, whereas the equilibrium point (0, 0) is stable. In addition, we can still know that there exists a continuum of equilibria on the edge of CD. Similar to the special case $\sigma =q\gamma$ in “Appendix A”, these equilibria are stable, but not asymptotically stable. Correspondingly, the equilibrium point (1, 0) is stable on the edge of CD. Furthermore, we know that $P_{D}>P_{C}$ in the interior phase space. Therefore, the equilibrium point (0, 1) can be the eventually evolutionary outcome.