Stochastic Stability in Three-Player Games with Time Delays

We discuss combined effects of stochasticity and time delays in finite-population three-player games with two mixed Nash equilibria and a pure one. We show that if basins of attraction of the stable interior equilibrium and the stable pure one are equal, then an arbitrary small time delay makes the pure one stochastically stable. Moreover, if the basin of attraction of the interior equilibrium is bigger than the one of the pure equilibrium, then there exists a critical time delay where the pure equilibrium becomes stochastically stable.


Introduction
Many social and biological processes can be modeled as systems of interacting individuals within the framework of evolutionary game theory [23,39,38,16,5,28,33]. Although in many models the number of players is very large, their strategic interactions are usually decomposed into a sum of two-player games. However, truly multi-player games naturally appear in many situations. For example, Haigh and Canning [15] discussed a multi-player War of Attrition, Pacheco et al. [30] analyzed a multi-player Stag Hunt game, Souza et al. [35] and Santos et al. [34] discussed a multi-player Snowdrift game. There have also appeared some systematic studies of multi-player games. Broom et al. [2] defined evolutionarily stable strategies for multi-player games and analyzed their properties, Kim [21] investigated an asymptotic and stochastic stability of Nash equilibria in multi-player games, Bukowski and Miȩkisz [3] provided a classification of symmetric three-player games with two strategies, fixation probabilities were discussed by Gokhale and Traulsen [10], see also [11,12].
For certain payoff parameters, such games have multiple evolutionarily stable strategies. For example, in one class of three-player games, we have one pure and one mixed evolutionarily stable strategy. We are faced with a standard problem of equilibrium selection. We will approach this problem from a dynamical point of view.
It is usually assumed that interactions between individuals take place instantaneously and their effects are immediate. In reality, all social and biological processes take a certain amount of time. It is natural therefore to introduce time delays into evolutionary games. It is well known that time delays may cause oscillations in solutions of ordinary differential equations [14,13,22,6,7]. Effects of time delay in evolutionary games were discussed in [36,29,1,37,17,18,27]. It was shown there that for certain models and time delays (above a critical value where the Hopf bifurcation appears), evolutionary dynamics exhibits oscillations and cycles, and interior equilibria ceased to be asymptotically stable in discrete and continuous replicator dynamics. In particular, Moreira et al. [27] discussed multiplayer Stag Hunt game with time delays.
Replicator dynamics describe time evolution of frequencies of strategies in the limit of an infinite number of individuals. However, real populations are finite. Stochastic effects connected with random matching of players, mistakes of players, and biological mutations can play a significant role in such systems. Therefore, to describe and analyze their time evolution, one should use stochastic modeling.
For symmetric games with two strategies, a state of the population is given by the number of individuals playing, say, the first strategy. The selection part of the dynamics ensures that if the average payoff of a given strategy is bigger than the average payoff of the other one, then the number of individuals playing the given strategy increases. In the model introduced by Kandori et al. [20], one assumes (as in the standard replicator dynamics) that individuals receive average payoffs weighted by fractions of different strategies present in the population. Players may mutate with a small probability, hence the population may move against a selection pressure. To describe the long-run behavior of such stochastic dynamics, Foster and Young [8] introduced a concept of stochastic stability.
Here we will study how time delays affect stochastic stability of Nash equilibria of evolutionary games. In Section 2, we review the simple stochastic dynamics of a three-player game with two interior Nash equilibria. In Section 3, we present results concerning stochastic stability in the presence of time delays. Discussion follows in Section 4.

Stochastic evolutionary models in finite populations
We consider symmetric three-player games with two strategies, A and B, given by the following payoff matrices: where the left matrix gives payoffs for the row player, when the third player uses A, whereas the right matrix provides payoffs in the case of the third individual playing B.
In the well-mixed infinite populations, the expected values (with respect to the fraction of strategies in the population) of the payoffs of strategies are given by where x is the frequency of players with the A strategy. Mixed Nash equilibria are given by solutions of the equation f A = f B . The standard replicator dynamics reads [39,16]: The classification of of three-player games with respect to the number of Nash equilibria and evolutionarily stable strategies was provided in [3]. Here we consider games with three Nash equilibria: two mixed ones (an asymptotically stable x 1 and an unstable x 2 ) and an asymptotically stable pure one, x 3 = 1, as it is illustrated in a phase portrait of the replicator dynamics, see Fig.1.
To study the effects of stochastic perturbations on the stability of Nash equilibria we will deal with finite-population models. Namely, let us assume that our population consists of N individuals. The state of the population at any discrete time t is characterized by the number of individuals, z(t), playing the strategy A. Now to avoid unnecessary (and non-essential) technicalities, we will choose N and payoffs such that the Nash equilibria are given by natural numbers and are equal to Nash equilibria in infinite populations. It means that we allow self-interactions, The classical Kandori-Mailath-Rob evolutionary dynamics is described by the following rule [20]: with the probability 1 − and with the probability the population moves in the other direction in the first two cases; if f A (z t ) = f B (z t ), then the number of A-strategists stays the same with the probability 1− and decreases or increases by one at time t+1 with the probability /2; if z t = 0, then z t+1 = 0 with the probability 1 − and with the probability , z t+1 = 1; if z t = N , then z t+1 = N with the probability 1 − and with the probability , z t+1 = N − 1.
We have obtained an ergodic Markov chain with the unique stationary probability distribution -stationary state µ . It is easy to see that there are four absorbing sets of our dynamics with = 0: two interior states, z 1 and z 2 , with coexisting strategies, and two homogeneous ones, z 3 = N and z = 0, the last one is not a Nash equilibrium. Now the question is which absorbing states survive small stochastic perturbations; that is, which are in the support of the zero-limit of µ . The following concept of stochastic stability was introduced in [8].
Definition A state Y of a Markov chain with the unique stationary probability distribution µ is stochastically stable if lim →0 µ (Y ) = 1.
It means that along almost any trajectory, for a small mutation level , the frequency of visiting the state Y is close to 1.
It is clear that in our models the only candidates for stochastically stable states are the asymptotically stable (for = 0) absorbing states z 1 and z 3 . We may also intuitively expect that if the number of steps (mutations or mistakes which happen with the probability ) to get out of the basin of attraction of a given state is bigger than the number of steps to get out of the basin of attraction of the other state, then the given state is stochastically stable. The formal proof uses the tree lemma -the special representation of a stationary distribution of an ergodic Markov chain [9], see the Appendix.
In our paper we discuss two particular examples.
Proof : For every state there is only one rooted tree. In particular, it follows from the fact that z 2 − z 1 = N − z 2 that trees rooted at z 1 and z 3 have the same product of probabilities ( the system needs the same nmber of mutations to get out of basins of attraction of both states. Proposition follows from the tree lemma.

Proposition 2. z 1 is stochastically stable.
Proof : Now we have that z 2 − z 1 > N − z 2 . In particular, the tree rooted at z 1 has the leading term of the order 12 and that of z 3 has the order 8 . In other words, one needs 12 mistakes to get out of the basin of attraction of x 1 and 8 mistakes to get out of the basin of attraction of x 3 .
Let us mention that three-player games with random matching of players [32,24] were analyzed in [19]. Dependence of stochastic stability of equilibria on game parameters (payoffs) is much more complex there.

Stochastic models with time delays
Now we introduce a time delay τ into our stochastic evolutionary models of finite populations. Namely, let with the probability 1 − and with the probability the population moves in the other direction in the first two cases; if f A (z t−τ ) = f B (z t−τ ), then the number of A-strategists stays the same with the probability 1 − and decreases or increases by one at time t + 1 with the probability /2; if z t−τ = 0, then z t+1 = z t − 1 with the probability 1 − and with the probability , z t+1 = z t ; if z t = N , then z t+1 = z t + 1 with the probability 1 − and with the probability , z t+1 = z t − 1.
Let us note that because of the time delay, we have to specify initial conditions for all discrete moments of time −τ ≤ t ≤ 0. To restore the Markov property of our dynamics, we redefine states of our our system to be τ + 1 tuples (z t−τ , z t−τ +1 , ..., z t ) at time t. In that way we get a Markov chain with the unique stationary probability distribution. Similar dynamical models with transition probabilities depending upon the finite history are known as high order Markov chains [4,31].
Let us assume that τ < z 2 − z 1 . It is easy to see the cycle around z 1 with the amplitude τ and the time period 4τ + 2 is a trajectory of the deterministic part of the dynamics (3.1) that is it is invariant under the deterministic rule (3.1). Moreover, it was proven in [26] that when we start with any consistent initial condition (z 0 , z −1 , ..., z −τ ), that is |z t − z t−1 | ≤ 1, t = 0, ..., −τ + 1 and here we additionally assume that z 0 , z −1 , ..., z −τ < z 2 with not all z's equal to 0, we end up in the cycle in a finite time. It was also proven in [26] that the cycle is stochastically stable. Now let us observe that once we have oscillations around z 1 ,it is easier to escape the basin of attraction of the cycle. It is easy to see, counting the number of mistakes, that when z 1 < N − z 2 + τ , then z 3 = N becomes stochastically stable even if it was not stochastically stable in the case without time delay. We use the tree lemma and get following propositions for our models.  Let us note that the definition of stochastic stability involves two limits. For any fix but low we take the limit of t → ∞ to get the stationary probability distribution and then we take the limit → 0 for a sequence of dependent of stationary distributions. It is clear that for a very low , if we start with initial conditions close to non-stochastically stable absorbing state (or cycle) we might have long time to converge to the vicinity of the stochastically stable state (one needs many mistakes and each of them has the probability ). Results of stochastic simulations for both models and various time delays are presented in Figs. 4-5. As expected, we see that bigger the time delay, smaller is the time, the population needs to arrive in the neighborhood of a stochastically stable state.

Discussion
It is well known that time delays may cause oscillations in dynamical systems. In discrete systems, asymptotically stable states may loose their stability for any time delays. Here we discussed finite populations with a simple stochastic dynamics. More precisely, we studied three-player games with two inerior Nash equilibria (one stable and one unstable) with coexisting strategies, and a stable homogeneous one, in fixed-size populations. We showed that if basins of attraction of the stable interior equilibrium and the stable homogeneous one are equal, then an arbitrary small time delay makes the homogeneous one stochastically stable. The reason is that in the presence of a time delay, the interior equilibrium looses its stability -there appears a stable cycle -and then it is easier to get out of its basin of attraction by stochastic perturbations. Moreover, if the basin of attraction of the interior equilibrium is bigger than the one of the homogeneous equilibrium, then there exists a critical time delay where the homogeneous equilibrium becomes stochastically stable, even though it is unstable for the zero time delay.
We would like to emphasize that global stability of the homogeneous equilibrium in our model is a combined effect of both stochasticity and time delays. In the absence of a time delay, the interior equilibrium is stochastically stable, while in the absence of stochastic perturbations, both the homogeneous equilibrium and the cycle around the interior equilibrium are locally asymptotically stable.
It is important to study combined effects of time delays and stochasticity in more complex evolutionary systems. The work is in progress.

Stationary probability distributions of ergodic Markov chains
The following tree representation of a unique stationary probability distribution of an ergodic Markov chain was proposed in [9].
Let (Ω, P ) be an ergodic Markov chain with a state space Ω and transition probabilities given by P : Ω × Ω → [0, 1]. It has the unique stationary probability distribution µ . For x ∈ Ω, an x-tree is a directed graph on Ω (connecting all vertices) such that from every y = x there is a unique path to x and there are no outcoming edges out of x. Denote by T (x) the set of all x-trees and let q (x) = d∈T (x) (y,y )∈d P (y, y ), (4.1) where P (y, y ) is the element of the transition matrix (that is, a conditional probability that the system will be at the state y at time t + 1 provided it was at state y at time t) and the above product is with respect to all edges of the x−tree d. Here in our paper we assumed that the system follows some deterministic rule with the probability 1 − and with the probability , a mistake is made that moves the system in the other direction hence P (y, y ) is equal either to 1 − , , or 0. Now one can show that for all x ∈ Ω. It follows from that the stationary probability distribution can be written as the ratio of two polynomials in . Hence any non-absorbing state (for = 0) has zero probability in the stationary distribution in the zero-limit. Moreover, in order to study the zero-limit of the stationary distribution, it is enough to consider paths between absorbing states. Assume for example, like in our models, that we have two absorbing states (sets): x and y. Let m xy be a minimal number of mutations (mistakes) needed to make a transition from the state x to y and m yx the minimal number of mutations to make a transition from y to x. Then q (x) is of the order myx and q (y) is of the order mxy . If for example, m yx < m xy , then it follows that lim →0 µ (x) = 1, hence x is stochastically stable.