On the Computation of Value Correspondences for Dynamic Games

Abstract

Recursive game theory provides theoretic procedures for computing the equilibrium payoff or value sets of repeated games and the equilibrium payoff or value correspondences of dynamic games. In this paper, we propose and implement outer and inner approximation methods for equilibrium value correspondences that naturally occur in the analysis of dynamic games. The procedure utilizes set-valued step functions. We provide an application to a bilateral insurance game with storage.

This is a preview of subscription content, log in to check access.

Fig. 1

Notes

  1. 1.

    See Atkeson [3], Phelan and Stacchetti [14], Sleet [15], and Fernandez-Villaverde and Tsyvinski [10] for applications to international lending and macroeconomic policy.

  2. 2.

    The usefulness of inner and outer approximations in providing error bounds was first emphasized by Judd et al. [12] in the context of repeated games.

  3. 3.

    See Chang [7] for an application of discrete methods to a repeated policy game. See Phelan and Stacchetti [14] for an application of linear interpolation methods to a dynamic policy game.

  4. 4.

    Thus, we allow for an uncountable number of endogenous states. The restriction to a finite set \(S\) avoids measure-theoretic complications.

  5. 5.

    For a correspondence \(C:S\times K\rightrightarrows X\), Graph \(C(s)=\{(k,x):x\in C(s,k)\}\) and Graph \(C=\{(s,k,x):x\in C(s,k)\}\).

  6. 6.

    However, each Graph \(V^{L*}(s)\) is not generally convex.

  7. 7.

    Geometrically, for \(k\in K_q\), \(\omega ^{\mathbb O}_{s,q}(k)\) equals the intersection of a finite number of half-spaces containing and with hyperplanes tangent to co \(\cup _{k^{\prime }\in K_{q}}V(s,k^{\prime })\).

  8. 8.

    The stronger assumption of a non-empty interior is needed in this case.

  9. 9.

    The surface of the step correspondence contains discontinuities at the joins of the partition elements. Application of a nonlinear optimizer requires either that the correspondence is smoothed across these joins, or that the optimizations are split into a collection of sub-optimizations in each of which the continuation state variable is restricted to an element of the partition.

References

  1. 1.

    Abreu D, Pearce D, Stacchetti E (1986) Optimal cartel equilibria with imperfect monitoring. J Econ Theory 39(1): 251–269

  2. 2.

    Abreu D, Pearce D, Stacchetti E (1990) Toward a theory of discounted repeated games with imperfect monitoring. Econometrica 58(5):1041–1063

  3. 3.

    Atkeson A (1991) International lending with moral hazard and risk of repudiation. Econometrica 59:1069–1089

    MathSciNet  Article  MATH  Google Scholar 

  4. 4.

    Balbus L, Reffett K, Wozny L (2014) A constructive study of markov equilibria in stochastic games with strategic complementarities. J Econ Theory 150:815–840

    MathSciNet  Article  MATH  Google Scholar 

  5. 5.

    Baldauf M, Judd K, Mollner J, Yeltekin Ş (2013) Supergames with states. Stanford University, Mimeo

    Google Scholar 

  6. 6.

    Beer G (1980) The approximation of upper semicontinuous multifunctions by step multifunctions. Pac J Math 87:11–19

    MathSciNet  Article  MATH  Google Scholar 

  7. 7.

    Chang R (1998) Credible monetary policy in an infinite horizon model: recursive approaches. J Econ Theory 81:431–461

    MathSciNet  Article  MATH  Google Scholar 

  8. 8.

    Cronshaw M, Luenberger D (1994) Strongly symmetric subgame perfect equilibrium in infinitely repeated games with perfect monitoring and discounting. Games Econ Behav 6:220–237

    MathSciNet  Article  MATH  Google Scholar 

  9. 9.

    Feng Z, Miao J, Peralta-Alva A, Santos M (2014) Numerical simulation of nonoptimal dynamic equilibrium models. Int Econ Rev 55(1):83–110

    MathSciNet  Article  MATH  Google Scholar 

  10. 10.

    Fernandez-Villaverde J, Tsyvinski A (2002) Optimal fiscal policy in a business cycle model without commitment. University of Pennsylvania, Mimeo

    Google Scholar 

  11. 11.

    Herings PJ-J, Peeters R (2004) Stationary equilibria in stochastic games: structure, selection and computation. J Econ Theory 118:32–60

    MathSciNet  MATH  Google Scholar 

  12. 12.

    Judd K, Yeltekin Ş, Conklin J (2003) Computing supergame equilibria. Econometrica 71:1239–1255

    MathSciNet  Article  MATH  Google Scholar 

  13. 13.

    Kocherlakota N (1996) Efficient bilateral risk sharing without commitment. Rev Econ Stud 63:595–609

    Article  MATH  Google Scholar 

  14. 14.

    Phelan C, Stacchetti E (2001) Subgame perfect equilibria in a ramsey taxes model. Econometrica 69:1491–1518

    MathSciNet  Article  MATH  Google Scholar 

  15. 15.

    Sleet C (1996) New recursive methods for government policy problems. Stanford University, Mimeo

    Google Scholar 

  16. 16.

    Yeltekin Ş, Judd K, Cai Y (2014) Computing equilibria of dynamic games. Working Paper. Carnegie Mellon University

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Christopher Sleet.

Appendix

Appendix

The following algorithms provide the details of the computational procedure for the outer and inner monotone approximations of \(B(V)(s,k)\) using hyperplanes. The correspondence \(V(s,k)\) is a candidate for the equilibrium value correspondence.

______________________________________

Outer Approximation Algorithm

  1. 1.

    Fixed Inputs:

    1. (a)

      Exogenous parameters.

    2. (b)

      Partition \(\mathcal {K} = \{K_q\}\).

    3. (c)

      Search directions: \({\mathcal {R}}=\{r_1,\ldots , r_{\overline{\ell }_r} \}\), with each \(\{r_{\ell }\in {\mathbb {R}}^I \ : \ ||r_{\ell }|| =1\}\).

    4. (d)

      Convergence criterion \(\varepsilon >0\).

  2. 2.

    First Iteration Inputs:

    1. (a)

      Set of boundary points \(\{v_{\ell }(s,q)|\ell = 1,\ldots , \overline{\ell }\}\) for each \((s,q)\).

    2. (b)

      Set of corresponding normals: \({\mathcal {Z}}=\{z_{1},\ldots ,{z}_{\overline{\ell }}\}\). Together (a) and (b) define the initial outer approximation \(V^{\mathbb {O}}(s,k)=\cup _{q} \omega ^{\mathbb {O}}_{s,q}(k)\) where

      $$\begin{aligned} \omega ^{\mathbb {O}}_{s,q}(k) = {\left\{ \begin{array}{ll} \cap _{\ell =1}^{\overline{\ell }}\{v\;|\;z_{\ell }\cdot v\le z_{\ell }\cdot v_\ell (s,q)=c_{\ell }(s,q)\}, &{} \text {if}\,k \in K_{q}, \\ \emptyset &{} \text {otherwise}. \end{array}\right. } \end{aligned}$$

      for all \(v\in \) co \( \cup _{k^{\prime }\in K_{q}}V(s,k^{\prime })\).

  3. 3.

    Optimization: Compute the new subgradient and boundary point sets that together represent an outer approximation to \(B^L({V})\). For each \(s \in S\), each partition element \(K_{q}\) and each \({r_{\ell }}\in {{\mathcal {R}}}\)

    $$\begin{aligned} c_{\ell }^{+}(s,q)&= {\max _{a,v^{\prime },k \in K_{q}}}\ r_{\ell }\cdot [(1-\beta )u(s,k,a)+ \beta \sum _{S} v^{\prime }(s^{\prime })\Pi (s^{\prime }|s)] \;\; \text {s.t.} \\&(i)\;v^{\prime }(s^{\prime })\in V^{\mathbb {O}}(s^{\prime },f(s,k,a)), \ a \in A(s,k), \nonumber \\&(ii)\;\text {for each } i \in \{1,\ldots ,I\}, \nonumber \\&\;\;\;\;\ (1-\beta )u^{i}(s,k,a)+\beta \sum _{S} v^{i\prime }(s^{\prime })\Pi (s^{\prime }|s)\ge \nonumber \\&\;\;\;\;\; \;\;\;\;\;\;\;\;\; \max _{a_{-i}\in A^{i}(s,k)} \min _{\tilde{v}^i(s^{\prime })\in V^{\mathbb {O}}(s^{\prime },f(s,k,a_{-i},a^{\prime }))} \ \ (1-\beta )u^{i}(s,k,a_{-i},a^{\prime })\nonumber \\ {}&+\,\, \beta \sum _{S}\tilde{v}^{i}(s^{\prime })\Pi (s^{\prime }|s). \nonumber \end{aligned}$$
    (6)
  4. 4.

    Output: \(V^{+}(k,s)={\hat{B}}(V)(s,k)=\cup _{q}\omega _{s,q}^{{\mathbb {O}}^{+}}(k)\), where

    $$\begin{aligned} \omega _{s,q}^{{\mathbb {O}}^{+}}(k) = {\left\{ \begin{array}{ll} \cap _{\ell =1}^{\overline{\ell }_{r}}\{v\;|\;r_{\ell }\cdot v\le c_{\ell }^{+}(s,q)\} &{} \text {if}\,k \in K_{q}, \\ \emptyset &{} \text {otherwise.} \end{array}\right. } \end{aligned}$$
  5. 5.

    Check for convergence: Stop if \(d_{V} ({V}^{+}, {V}) < \varepsilon \). Else set \({V}^{\mathbb {O}}={V}^{+}\), \({\mathcal {Z}}=R\), and \(\overline{\ell } = \overline{\ell }_r\) and return to Step 3.

______________________________________

The main computational work is done in the optimization step. In this step, for each search direction \(r_{\ell }\) and element of the partition \(\mathcal {K}\), we find an action profile \(a\), continuation values \(v^{\prime }(s^{\prime }) \in V(s^{\prime },f(s,k,a))\), and current state \(k\) in \(K_{q}\) which makes \(a\) incentive compatible and maximizes a weighted sum of player payoffs where the weights are given by \(r_{\ell }\).Footnote 9

In the outer approximation case, the set of search directions, \({{\mathcal {R}}}\), and the set of hyperplane normals for the approximation of \(V(s,k)\), \({\mathcal {Z}}\), coincide after the initial iteration. Correspondences are represented by sets of bounding hyperplanes, one set for each block in the partition and each state \(s\). In this way, an approximated correspondence can be reduced to a collection \( \{c_{\ell }(s,q)\}\) in \({\mathbb {R}}^{SQI\overline{\ell }}\), with each \(c_{\ell }(s,q)\) determining the position of the \(\ell \)-th hyperplane on the \(q\)-th block in state \(s\).

For inner approximations, the set of hyperplane normals vary from iteration to iteration, depending on the number and position of the points on the convex hull of \(V(s,k)\). The bounding hyperplanes are constructed from the line segments that connect the extreme points belonging to the convex hull of \(V(s,k)\). In this case, an approximated correspondence cannot be reduced to a collection of \(\{c_{\ell }(s,q)\}\) in \({\mathbb {R}}^{SQI\overline{\ell }}\) but can be reduced to a collection of \(\{v_{\ell }(s,q)\}\) in \(\cap _{k^{\prime }\in K_{q}}V(s,k^{\prime })\) for each \((s,q)\). This requires a slight modification to the output and updating used in the outer approximation algorithm. Below is the step by step description of the algorithm for computing an inner approximation to \(B(V)\).

______________________________________

Inner Approximation Algorithm

  1. 1.

    Fixed Inputs:

    1. (a)

      Exogenous parameters

    2. (b)

      Partition \({\mathcal {K}} = \{K_q\}\).

    3. (c)

      Search directions: \({{\mathcal {R}}}=\{r_1,\ldots , r_{\overline{\ell }_r} \}\), with each \(\{r_{\ell } \in {\mathbb {R}}^I \ : \ ||r_{\ell }|| =1\}\).

    4. (d)

      Convergence criterion: \(\varepsilon >0\).

  2. 2.

    First Iteration Inputs:

    1. (a)

      Set of points \(\{v_{\ell }(s,q)|\ell = 1,\ldots , \overline{\ell }\}\) in \( \cap _{k^{\prime }\in K_{q}}V(s,k^{\prime })\), for each \((s,q)\). These points define the initial inner approximation \(V^{\mathbb {I}}(s,k)=\cap _{q} \omega ^{\mathbb {I}}_{s,q}(k)\) where

      $$\begin{aligned} \omega ^{\mathbb {I}}_{s,q}(k) = {\left\{ \begin{array}{ll} \text {co} \ \{v_{\ell }(s,q)|\ell = 1,\ldots , \overline{\ell }\} &{} \text {if}\,k \in K_{q}, \\ {\mathbb {R}}^{I} &{} \text {otherwise}. \end{array}\right. } \end{aligned}$$
  3. 3.

    Optimization: Compute the new sets of point from which the convex hulls that together represent an inner approximation to \(B^L({V})\) are constructed. For each \(s \in S\), each partition element \(K_{q}\) and each \({r_{\ell }}\in {{\mathcal {R}}}\)

    $$\begin{aligned} \{a,v^{\prime },k\}_{\ell }^{+}(s,q)&= \mathop {{{\mathrm{argmax}}}}\limits _{a,v^{\prime },k \in K_{q}}\;\;\; r_{\ell }\cdot [(1-\beta )u(s,k,a)+ \beta \sum _{S} v^{\prime }(s^{\prime })\Pi (s^{\prime }|s)] \ \text {s.t} \\&(i)\;v^{\prime }(s^{\prime })\in V^{\mathbb {I}}(s^{\prime },f(s,k,a)), \ a \in A(s,k), \nonumber \\&(ii)\;\text {for each } i \in \{1,\ldots ,I\}, \nonumber \\&\;\;\;\;\ (1-\beta )u^{i}(s,k,a)+\beta \sum _{S} v^{i\prime }(s^{\prime })\Pi (s^{\prime }|s)\ge \nonumber \\&\;\;\;\;\; \ \max _{a_{-i}\in A^{i}(s,k)} \min _{\tilde{v}^i(s^{\prime })\in V^{\mathbb {I}}(s^{\prime },f(s,k,a_{-i},a^{\prime }))} \ (1-\beta )u^{i}(s,k,a_{-i},a^{\prime })\nonumber \\&\quad +\, \beta \sum _{S}\tilde{v}^{i}(s^{\prime })\Pi (s^{\prime }|s). \nonumber \end{aligned}$$
    (7)
  4. 4.

    Output: \(V^{+}(k,s)={\hat{B}}(V)(s,k)=\cap _{q}\omega _{s,q}^{{\mathbb {I}}^{+}}(k)\), with

    $$\begin{aligned} \omega _{s,q}^{{\mathbb {I}}^{+}}(k) = {\left\{ \begin{array}{ll} \text {co\ } \{v^{+}_{\ell }(s,q)|\ell = 1,\ldots , \overline{\ell }\} &{} \text {if}\,k \in K_{q}, \\ {\mathbb {R}}^{I} &{} \text {otherwise.} \end{array}\right. } \end{aligned}$$

    where \(v^{+}_{\ell }(s,q)\) is equal to \([(1-\beta )u(s,k,a)+ \beta \sum _{S} v^{\prime }(s^{\prime })\Pi (s^{\prime }|s)]\) evaluated at \(\{a,v^{\prime },k\}_{\ell }^{+}(s,q)\).

  5. 5.

    Check for convergence: Stop if \(d_V ({V}^{+}, {V}) < \varepsilon \). Else set \({V}^{\mathbb {I}}(s,k)={V}^{+}(s,k)\) and go back to Step 3.

See Judd et al. [12] and Yeltekin et al. [16] for the details of constructing convex hulls similar to those required in Step 4.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Sleet, C., Yeltekin, Ş. On the Computation of Value Correspondences for Dynamic Games. Dyn Games Appl 6, 174–186 (2016). https://doi.org/10.1007/s13235-015-0139-1

Download citation

Keywords

  • Dynamic games
  • Computation
  • Approximation
  • Value correspondences