Stationary, completely mixed and symmetric optimal and equilibrium strategies in stochastic games

Abstract

In this paper, we address various types of two-person stochastic games—both zero-sum and nonzero-sum, discounted and undiscounted. In particular, we address different aspects of stochastic games, namely: (1) When is a two-person stochastic game completely mixed? (2) Can we identify classes of undiscounted zero-sum stochastic games that have stationary optimal strategies? (3) When does a two-person stochastic game possess symmetric optimal/equilibrium strategies? Firstly, we provide some necessary and some sufficient conditions under which certain classes of discounted and undiscounted stochastic games are completely mixed. In particular, we show that, if a discounted zero-sum switching control stochastic game with symmetric payoff matrices has a completely mixed stationary optimal strategy, then the stochastic game is completely mixed if and only if the matrix games restricted to states are all completely mixed. Secondly, we identify certain classes of undiscounted zero-sum stochastic games that have stationary optima under specific conditions for individual payoff matrices and transition probabilities. Thirdly, we provide sufficient conditions for discounted as well as certain classes of undiscounted stochastic games to have symmetric optimal/equilibrium strategies—namely, transitions are symmetric and the payoff matrices of one player are the transpose of those of the other. We also provide a sufficient condition for the stochastic game to have a symmetric pure strategy equilibrium. We also provide examples to show the sharpness of our results.

Appendix

The proofs of Theorems 7 and 9 are published in the Proceedings of a Conference that are not available online, and is hence not easily accessible currently. For the sake of completeness, we provide these proofs below.

We first provide necessary notation and known results that are used in the proofs. For the bimatrix game (A, B), let $\varepsilon$ denote the set of equilibrium strategies. Let $(x_0, y_0) \in \varepsilon$. Further let Cofactor$(A_{ij})$ and Cofactor$(B_{ij})$ denote the cofactors of $a_{ij}$ and $b_{ij}$ respectively. For showing the conditions for a stochastic game to be completely mixed using the concept of principal Pfaffians, we need the following results by Oviedo (1996).

• 1. (Theorem 1 of Oviedo 1996) If $\varepsilon$ is completely mixed and $x_0^tAy_0 = x_0^tBy_0 = 0$, then there exists an i $(1 \le i \le n)$ such that Cofactor$(A_{i1})$, Cofactor$(A_{i2})$, ..., Cofactor$(A_{in})$ are different from zero and have the same sign. Similarly, there exists a j $(1 \le j \le n)$ such that Cofactor$(B_{j1})$, Cofactor$(B_{j2})$, ..., Cofactor$(B_{jn})$ are different from zero and have the same sign.

• 2. (Corollary 1 of Oviedo 1996) If $\varepsilon$ is completely mixed, then $x_0^tAy_0 = \frac{det (A)}{\sum \limits _{i, j} A_{ij}}$ and $x_0^tBy_0 = \frac{det (B)}{\sum \limits _{i, j} B_{ij}}$, where the denominators are always different from 0.

• 3. (Proposition 1 of Oviedo 1996) Suppose there exists constants $v_1$ and $v_2$ such that for any $(x, y) \in \varepsilon$, $Ay = v_1e$ and $x^tB = v_2e^t$. Suppose, moreover, that both A and B are square matrices of rank $n - 1$. Then $\varepsilon$ is completely mixed.

Proof of Theorem 7

(Theorem 3 of Sujatha et al. 2014) Let the set of all equilibrium strategies $\varepsilon$ be completely mixed. Let x and y be the strategies used by player-1 and player-2 respectively. Since the matrices are odd ordered skew symmetric, $det (A) = det (B) = 0$. Then, $x^tAy = x^tBy = 0$ by Oviedo (1996, Corollary 1).

Further by Oviedo (1996, Theorem 1) and without loss of generality, we can assume Cofactor$(A_{ij}) > 0$ for all i, for all j. That is, $(-1)^{i+j} m_{ij} > 0$ where $m_{ij}$ is the sub-determinant obtained by deleting the ith row and jth column of the matrix A. By Kaplansky (1995, Theorem 1), this implies that $(-1)^{i+j} p_{i}p_{j} > 0$ where $p_{i}$ and $p_{j}$ are the i$^\mathrm{th}$ and j$^\mathrm{th}$ principal Pfaffians of A.

If i and j are both even or both odd, then the above equation implies that either both $p_{i}$ and $p_{j}$ are greater than 0, or both are lesser than 0. If i is even and j is odd or vice versa, then either ($p_{i} > 0$, $p_{j} < 0$) or ($p_{i} < 0$, $p_{j} > 0$). Thus all the principal Pfaffians of A are nonzero and alternate in sign. The same holds for all the principal Pfaffians of matrix B.

Conversely, let the principal Pfaffians of matrix A be nonzero and alternate in sign. Without loss of generality, we can assume that $p_i > 0$ where i is odd and $p_j < 0$ where j is even.

Then, $(-1)^{i+j}p_{i}p_{j} > 0$. That is, $(-1)^{i+j}m_{ij} > 0$, where $m_{ij}$ is the sub-determinant obtained by deleting the ith row and jth column of matrix A. This implies that $c_{ij} > 0$, where $c_{ij}$ is the cofactor of $a_{ij}$.

Hence all cofactors of A (and similarly of B) are nonzero and have the same sign. The rank of both these matrices is $n-1$ since the minors of order $n-1$ are nonzero. Consider a strategy $(x_0, y_0) \in \varepsilon$. Then there exists $v_1$ and $v_2$ such that $Ay_0 = v_1e$ and $(x_0)^tB = v_2e^t$. By Oviedo (1996, Proposition 1), $\varepsilon$ is completely mixed. $\square$

Proof of Theorem 9

(Theorem 5 of Sujatha et al. 2014) Let $f^o (s)$ be the optimal strategy for player-1 for R(s). Gale (1960) showed that for a finite zero-sum game with a skew symmetric matrix, the value of the game is 0 and any strategy that is optimal for one player is also optimal for the other player. Hence, if we view R(s) as a finite zero sum game with skew symmetric payoff matrix, then the value of the stochastic game is 0. Let g be any strategy for player-2. We have $r (s, f^o, g) \ge 0$.

Indicate r(f, g) by $\begin{pmatrix} r(1, \quad f(1), \quad g(1))\\ r(2,\quad f(2), \quad g(2))\\ \vdots \\ r(N, \quad f(N),\quad g(N)) \end{pmatrix}.$

The total expected $\beta$-discounted income for player-1 is given by

where $Q (f^o, g)$ is a $N \times N$ matrix with $(s, s')$th element representing the transition probability $q (s' | s, f^o, g)$. It is obvious that for all g, for all s

$$\begin{aligned} I_\beta (f^o, g) (s)\ge 0 \end{aligned}$$

(5)

Similarly, let $g^o$ be an optimal stationary strategy for player-2. The total expected $\beta$-discounted income for player-2 is given by

(6)

By Gale’s result (1960), any strategy that is optimal for one player is also optimal for the other player, that is $g^o = f^o$. Hence, $I_\beta (f, f^o)(s) = [I - \beta Q (f, f^o)]^{-1} r (s, f, f^o)$ (from Eq. 6). Hence for all s, for all f,

$$\begin{aligned} I_\beta (f, f^o)(s) \le 0 \end{aligned}$$

(7)

Comparing Eqs. 5 and 7, we have $I_\beta (f^o, g)(s) = I_\beta (f, f^o)(s) = 0$. That is, the value of the stochastic game starting in state s is 0. Note that the auxiliary game and the matrix game restricted to state s coincide for all discount factors. Thus the stochastic game has symmetric optimal stationary strategies independent of the discount factor $\beta$ and the transition probability $q (s' | s, f, g)$. $\square$