Discounted stochastic games with voluntary transfers

Abstract

This paper studies discounted stochastic games with perfect or imperfect public monitoring and the opportunity to conduct voluntary monetary transfers and possibly burn money. This generalization of repeated games with transfers is ideally suited to study relational contracting in applications with long-term investments and also allows to study collusive industry dynamics. We show that for all discount factors every perfect public equilibrium payoff can be implemented with a class of simple equilibria that have a stationary structure on the equilibrium path and optimal penal codes with a stick-and-carrot structure. We develop an algorithm for perfect monitoring to compute the set of equilibrium payoffs and find simple equilibria that implement these payoffs.

Appendix: Remaining proofs

Proof of Theorem 2:

For each state $x\in X$ and regime $k\in \mathcal {K},$ condition (16) allows to choose a distribution $u_{i}^{k}(x),$ $i=1,\ldots ,n$, of the surplus such that

$$\begin{aligned} \sum _{i=1}^{n}u_{i}^{k}(x)=(1-\delta )\Pi (x,\alpha ^{k})+\delta E[U|x,\alpha ^{k}] \end{aligned}$$

(30)

and

$$\begin{aligned} u_{i}^{k}(x)\ge \max _{\hat{a}_{i}}(1-\delta )\pi _{i}(x,\hat{a}_{i},\alpha _{-i}^{k})+\delta E[v_{i}|x,\hat{a}_{i},\alpha _{-i}^{k}], \end{aligned}$$

(31)

holding with equality for $i=k.$ A simple strategy profile with transfers $p_{i}^{k}(x,\alpha ^{k}(x),x')$ achieves this distribution of payoffs if the expected transfers

$$\begin{aligned} \bar{t}_{i}^{k}(x)=(1-\delta )E[p_{i}^{k}(x,\alpha ^{k}(x),x')|x,\alpha ^{k}(x)] \end{aligned}$$

satisfy

$$\begin{aligned} \delta \bar{t}_{i}^{k}(x)=(1-\delta )\pi _{i}(x,\alpha ^{k})+\delta E[u_{i}^{e}|x,\alpha ^{k}]-u_{i}^{k}(x). \end{aligned}$$

If we define $\bar{t}_{i}^{k}(x)$ by this condition, it holds that $\sum _{i=1}^{n}\bar{t}_{i}^{k}(x)=0$. Moreover, it follows from condition (31) that

$$\begin{aligned} E\left[ u_{i}^{e}-v_{i}|x,\alpha ^{k}\right] \ge \bar{t}_{i}^{k}(x). \end{aligned}$$

The intuition behind this is that it is more difficult to induce an action and a subsequent expected payment afterward than to induce an expected payment. We still need to show that for each $k\in \mathcal {K}$ and state x there exist payments $t_{i}(x')=(1-\delta )p_{i}^{k}(x,\alpha ^{k}(x),x')$ for each state $x'$ such that the following three conditions hold:

$$\begin{aligned}&\displaystyle t_{i}(x') \le u_{i}^{e}(x')-v_{i}(x'), \end{aligned}$$

(32)

$$\begin{aligned}&\displaystyle \sum _{i=1}^{n}t_{i}(x') = 0,\end{aligned}$$

(33)

$$\begin{aligned}&\displaystyle \sum _{_{x'}}q(x')t_{i}(x') = \bar{t}_{i}^{k}(x), \end{aligned}$$

(34)

where $q(x')=q(x'|x,\alpha ^{k}(x))$ is the transition probability from state x to state $x'$ if $\alpha ^{k}(x)$ is played. We use Theorem 22.1 in Rockafellar (1970) to show that such payments exist. This theorem says that the existence of a vector with entries $t_{i}(x')$, $i=1,\ldots ,n$, $x'\in X$, that satisfies the above three conditions is equivalent to the nonexistence of real numbers $\lambda _{i}(x')\ge 0$, $\mu (x'),$ and $\eta _{i},$ $i=1,\ldots ,n$, $x'\in X$, that satisfy the following two conditions:

$$\begin{aligned}&\displaystyle \lambda _{i}(x')+\mu (x')+\eta _{i}q(x') = 0\text { for all }i,x'\end{aligned}$$

(35)

$$\begin{aligned}&\displaystyle \sum _{i,x'}\lambda _{i}(x')(u_{i}^{e}(x')-v_{i}(x'))+\sum _{i=1}^{n}\eta _{i}\bar{t_{i}}^{k}(x) < 0. \end{aligned}$$

(36)

We assume to the contrary that such a solution to (35) and (36) exists. These two conditions imply that

$$\begin{aligned} -\sum _{x'}\mu (x')\left( U(x')-\sum _{i=1}^{n}v_{i}(x')\right) +\sum _{i=1}^{n}\eta _{i}\left( \bar{t_{i}}^{k}(x)-E[u_{i}^{e}-v_{i}|x]\right) <0. \end{aligned}$$

Let $\tilde{x}$ be a state with $\frac{\mu (\tilde{x})}{q(\tilde{x})}\le \frac{\mu (x')}{q(x')}$ for all $x'\in X.$ Since condition (35) holds for all $x'\in X,$ it also holds for $x'=\tilde{x},$ i.e., $\eta _{i}=-\frac{\lambda _{i}(\tilde{x})+\mu (\tilde{x})}{q(\tilde{x})}.$ Hence, it follows that

$$\begin{aligned}&\sum _{x'}\left( \frac{\mu (\tilde{x})q(x')}{q(\tilde{x})}-\mu (x')\right) \left( U(x')-\sum _{i=1}^{n}v_{i}(x')\right) \nonumber \\&\quad +\sum _{i=1}^{n}\frac{\lambda _{i}(\tilde{x})}{q(\tilde{x})}\left( E[u_{i}^{e}-v_{i}|x]-\bar{t_{i}}^{k}(x)\right) <0. \end{aligned}$$

This implies

$$\begin{aligned} \sum _{x'}\left( \frac{\mu (\tilde{x})q(x')}{q(\tilde{x})}-\mu (x')\right) \left( U(x')-\sum _{i=1}^{n}v_{i}(x')\right) <0. \end{aligned}$$

By definition of $\tilde{x}$ and because of condition (15), the expression on the left-hand side is nonnegative. Hence, we arrived at a contradiction, which means that the system given by (32), (33), and (34) must have a solution, and we can define payments $(1-\delta )p_{i}^{k}(x,\alpha ^{k}(x),x')=t_{i}(x')$.

It remains to define the payments following a unilateral deviation. For any combination of states x,$x^{\prime }$ and signal y with $y_{i}\ne \alpha _{i}^{k}(x)$ and $y_{-i}=\alpha _{-i}^{k}(x)$ we choose payments

$$\begin{aligned} (1-\delta )p_{i}^{k}(x,y,x^{\prime })=u_{i}^{e}(x^{\prime })-v_{i}(x^{\prime }), \end{aligned}$$

(37)

such that continuation payoffs after a deviation in the action stage are indeed given by $v_{i}$. Payments for players other than i can be defined such that

$$\begin{aligned} (1-\delta )p_{j}^{k}(x,y,x^{\prime })\le u_{j}^{e}(x^{\prime })-v_{j}(x^{\prime }) \end{aligned}$$

and

$$\begin{aligned} \sum _{j=1}^{n}p_{j}^{k}(x,y,x^{\prime })=0, \end{aligned}$$

using condition (15).

Now we have to show that the so defined simple strategy profile is indeed a PPE. The budget and payment constraints are satisfied by definition. The relevant action constraints are satisfied because of inequality (31). Moreover, it holds by definition that $u_{i}^{i}(x)=v_{i}(x),$ and since there is no money burning, $U^{e}(x)=U(x).$ The payment plan that we have defined is an optimal payment plan, because U(x) is an upper bound of the equilibrium joint payoff $U^{e}(x)$, and similarly

$$\begin{aligned} u_{i}^{i}(x)\ge \max _{a_{i}}(1-\delta )\pi _{i}(x,a_{i},\alpha _{-i}^{i})+\delta E[u_{i}^{i}|x,a_{i},\alpha _{-i}^{i}] \end{aligned}$$

implies that $v_{i}(x)$ is a lower bound of the punishment payoff $u_{i}^{i}(x).$ Thus, we have shown the “if” part of the Theorem. The “only if” part is straightforward. $\square$

Proof of Proposition 3:

For a given policy $\alpha ,$ let $C_{i}^{\alpha }$ be an operator mapping the set of punishment payoffs into itself defined by

$$\begin{aligned} C_{i}^{\alpha }(v_{i})[x]=c_{i}(x,\alpha (x),v_{i}) \end{aligned}$$

It can be easily verified that $C_{i}^{\alpha }$ is a contraction-mapping operator. It follows from the contraction-mapping theorem that player i’s best-reply payoffs are given by the unique fixed point of $C_{i}^{\alpha }$, which we denote by $v_{i}(\alpha )$. This means

$$\begin{aligned} v_{i}(\alpha )=C_{i}^{\alpha }(v_{i}(\alpha )) \end{aligned}$$

(38)

It is a well-known result that the operator $C_{i}^{\alpha }$ is monotone:

$$\begin{aligned} v_{i}\le \tilde{v}_{i}\Rightarrow C_{i}^{\alpha }(v_{i})\le C_{i}^{\alpha }(\tilde{v}_{i}) \end{aligned}$$

(39)

where $v_{i}\le \tilde{v}_{i}$ is defined as $v_{i}(x)\le \tilde{v}_{i}(x)\forall x\in X$. We denote by $[C_{i}^{\alpha }]^{k}$ the operator that applies k times $C_{i}^{\alpha }$ and define its limit by

$$\begin{aligned}{}[C_{i}^{\alpha }]^{\infty }=\lim _{k\rightarrow \infty }[C_{i}^{\alpha }]^{k}. \end{aligned}$$

The contraction-mapping theorem implies that $[C_{i}^{\alpha }]^{\infty }$ is well defined and transforms every payoff function v into the fixed point of $C_{i}^{\alpha }$, i.e.,

$$\begin{aligned}{}[C_{i}^{\alpha }]^{\infty }(v)=v(\alpha ) \end{aligned}$$

(40)

Furthermore, it follows from monotonicity of $C_{i}^{\alpha }$ that

$$\begin{aligned} C_{i}^{\alpha }(v_{i})\le v_{i}\Rightarrow [C_{i}^{\alpha }]^{\infty }(v_{i})\le v_{i} \end{aligned}$$

(41)

and

$$\begin{aligned} C_{i}^{\alpha }(v_{i})<v_{i}\Rightarrow [C_{i}^{\alpha }]^{\infty }(v_{i})<v_{i} \end{aligned}$$

(42)

where two payoff functions $u_{i}$ and $\tilde{u}_{i}$ satisfy $u_{i}<\tilde{u}_{i}$ if $u_{i}\le \tilde{u}_{i}$ and $u_{i}\ne \tilde{u}_{i}$.

We now show that for any two policies a and $\tilde{a}$ the following monotonicity results hold

$$\begin{aligned}&\displaystyle C_{i}^{\alpha }(v(\alpha ))=C_{i}^{\tilde{\alpha }}(v(\alpha )) \Rightarrow v(\alpha )=v(\tilde{\alpha }) \end{aligned}$$

(43)

$$\begin{aligned}&\displaystyle C_{i}^{\alpha }(v(\alpha ))>C_{i}^{\tilde{\alpha }}(v(\alpha )) \Rightarrow v(\alpha )>v(\tilde{\alpha }) \end{aligned}$$

(44)

$$\begin{aligned}&\displaystyle v(\alpha )\nleq v(\tilde{\alpha }) \Rightarrow C_{i}^{\alpha }(v(\alpha ))\nleq C_{i}^{\tilde{\alpha }}(v(\alpha )) \end{aligned}$$

(45)

We exemplify the proof for (44). It follows from (38), the left part of (44), (41) and (40) that

$$\begin{aligned} v(\alpha )=C_{i}^{\alpha }(v(\alpha ))>C_{i}^{\tilde{\alpha }}(v(\alpha ))\ge \left[ C_{i}^{\tilde{\alpha }}\right] ^{\infty }(v(\alpha ))=v(\tilde{\alpha }). \end{aligned}$$

Equation 43 and can be proven similarly. To prove (45), assume that there is some $\tilde{\alpha }$ with $C_{i}^{\alpha }(v)\le C_{i}^{\tilde{\alpha }}(v)$ but $\tilde{v}\ngeq v$. We find

$$\begin{aligned} v=C_{i}^{\alpha }(v)\le C_{i}^{\tilde{\alpha }}(v)\le \left( C_{i}^{\tilde{\alpha }}\right) ^{\infty }(v)=\tilde{v} \end{aligned}$$

which contradicts the assumption $\tilde{v}\ngeq v$.

Intuitively, these monotonicity properties of the cheating payoff operator are crucial for why the algorithm works. If one wants to find out whether a policy $\tilde{\alpha }$ can yield lower punishment payoffs for player i than a policy $\alpha$, one does not have to solve player i’s Markov decision process under policy $\tilde{\alpha }$. It suffices to check whether for some state x the cheating payoffs given policy $\tilde{\alpha }$ and punishment payoffs $v(\alpha )$ are lower than $v(\alpha )(x)$. If this is not the case for any admissible policy $\tilde{\alpha }$, then a policy $\alpha$ is an optimal punishment policy, in the sense that it minimizes player i’s punishment payoffs in every state.

The fixed point condition (38) of the value determination step and the policy improvement step (20) imply that $v^{r}=C_{i}^{\alpha ^{r}}(v^{r})\ge C_{i}^{\alpha ^{r+1}}(v^{r})$. We first establish that if

$$\begin{aligned} v^{r}=C_{i}^{\alpha ^{r}}(v^{r})=C_{i}^{\alpha ^{r+1}}(v^{r}). \end{aligned}$$

(46)

then we have $v_{i}^{r}=\hat{v}_{i}$. For a proof by contradiction, assume that condition holds for some r but that there exists a policy $\hat{\alpha }$ such that $v(\alpha ^{r})\nleq v(\hat{\alpha }),$ i.e., $\hat{\alpha }$ leads in at least some state x to a strictly lower best-reply payoff for player i than $\alpha ^{r}$. By (45) this would imply $C_{i}^{\alpha ^{r}}(v^{r})\nless C_{i}^{\tilde{\alpha }}(v^{r})$. This means that $\hat{\alpha }$ must also be a solution to the policy improvement step and since (46) holds, we then must have

$$\begin{aligned} C_{i}^{\alpha ^{r}}(v^{r})=C_{i}^{\hat{\alpha }}(v^{r}) \end{aligned}$$

However, (43) then implies that $v(\alpha ^{r})=v(\hat{\alpha })$, which contradicts the assumption $v(\alpha ^{r})\nleq v(\hat{\alpha })$. Thus, if the algorithm stops in a round R, we indeed have $v^{R}=\hat{v}_{i}$.

If the algorithm does not stop in round r, it must be the case that $v^{r}=C_{i}^{\alpha ^{r}}(v^{r})>C_{i}^{\alpha ^{r+1}}(v^{r})$. (44) then directly implies the monotonicity result $v^{r}>v^{r+1}$. The algorithm always stops in a finite number of rounds since the number of policies is finite and there are no cycles because of the monotonicity result. $\square$

Proof of Corollary 1:

Assume there exists a simple equilibrium with action plan $(\alpha _{2}^{k})_{k}$ and with optimal payment plan $(p^{k})_{k}$ such that joint payoffs are U (which implies that $p_{1}^{e}+p_{2}^{e}=0)$ and punishment payoffs are $v_{1}$ and $\bar{v}_{2}.$ Define

$$\begin{aligned} t^{k}(x,y,x')=\frac{\delta }{1-\delta }(u_{2}^{e}(x')-(1-\delta )p_{2}^{k}(x,y,x')-\bar{v}_{2}). \end{aligned}$$

The conditions (PC-k) imply that $t^{k}(x,y,x')\ge 0$ and

$$\begin{aligned} t^{k}(x,y,x')\le \frac{\delta }{1-\delta }(U(x')-v_{1}(x')-\bar{v}_{2}). \end{aligned}$$

Moreover, the conditions (AC-k),

$$\begin{aligned} \alpha _{2}^{k}(x)\in \arg \max _{\tilde{a}}(1-\delta )\pi _{2}(x,\tilde{a})+\delta E[u_{2}^{e}(x')-(1-\delta )p_{2}^{k}(x,y,x')|x,\tilde{a}], \end{aligned}$$

imply that $\alpha _{2}^{k}(x)\in \arg \max _{\tilde{a}}\pi _{2}(x,\tilde{a})+E[t^{k}|x,\tilde{a}]$.

For the other direction, assume that there exist payments $t^{k}(x,y,x')$ as in the Proposition and define

$$\begin{aligned} \tilde{u}_{2}^{e}(x)=(1-\delta )\pi _{2}(x,\alpha ^{e})+E[(1-\delta )t^{e}(x,y,x')+\delta \bar{v}_{2}|x,\alpha ^{e}] \end{aligned}$$

and

$$\begin{aligned} (1-\delta )p_{2}^{e}(x,y,x')=\tilde{u}_{2}^{e}(x')-\frac{1-\delta }{\delta }t^{e}(x,y,x')-\bar{v}_{2} \end{aligned}$$

as well as $p_{1}^{e}=-p_{2}^{e}$, such that $U^{e}(x)=U(x)$ and (BC-e) hold by definition, and $\tilde{u}_{2}^{e}(x)=u_{2}^{e}(x).$ Moreover, (PC-e) for the agent holds because

$$\begin{aligned} u_{2}^{e}(x')-(1-\delta )p_{2}^{e}(x,y,x')=\frac{1-\delta }{\delta }t^{e}(x,y,x')+\bar{v}_{2}\ge \bar{v}_{2}, \end{aligned}$$

and for the principal because

$$\begin{aligned} u_{1}^{e}(x')-(1-\delta )p_{1}^{e}(x,y,x')=U(x')-\frac{1-\delta }{\delta }t^{e}(x,y,x')-\bar{v}_{2}\ge v_{1}(x'). \end{aligned}$$

Further define

$$\begin{aligned} (1-\delta )p_{2}^{2}(x,y,x')=u_{2}^{e}(x')-\bar{v}_{2} \end{aligned}$$

and $p_{1}^{2}=-p_{2}^{2},$ such that $u_{2}^{2}(x)=\bar{v}_{2}$ and (BC-2) and (PC-2) for the agent hold by definition, and for the principal because $U(x')\ge v_{1}(x')+\bar{v}_{2}$. Finally, define

$$\begin{aligned} (1-\delta )p_{1}^{1}(x,y,x')=u_{1}^{e}(x')-v_{1}(x') \end{aligned}$$

and

$$\begin{aligned} (1-\delta )p_{2}^{1}(x,y,x')=u_{2}^{e}(x')-\frac{1-\delta }{\delta }t^{1}(x,y,x')-\bar{v}_{2}, \end{aligned}$$

such that $u_{1}^{1}(x)=v_{1}(x)$ and (PC-1) for the agent hold by definition, and for the principal for the same reason as (PC-e). The condition (BC-1) then holds as well, since it is equivalent to $U(x')-v_{1}(x')-\bar{v}_{2}\ge \frac{1-\delta }{\delta }t^{1}(x,y,x')$. The action constraints follow from condition (24) since the payments were defined such that the agent’s continuation payoff is equal to $\frac{(1-\delta )}{\delta }t$ plus a constant.

In case of perfect monitoring, any deviation is punished by withholding transfers while following prescribed play is rewarded by the maximum payment. If there is perfect monitoring in state x,  the condition therefore is

$$\begin{aligned} \pi _{2}(x,a_{2}^{k})+\frac{\delta }{1-\delta }E[U-v_{1}-\bar{v}_{2}|x,a^{k}]\ge \max _{\tilde{a}_{2}}\pi _{2}(x,\tilde{a}_{2}) \end{aligned}$$

which can be rearranged to equal condition (25). $\square$