Dynamic Games and Applications

, Volume 6, Issue 3, pp 304–323 | Cite as

Elementary Subpaths in Discounted Stochastic Games

  • Kimmo Berg


This paper examines the subgame-perfect equilibria in discounted stochastic games with finite state and action spaces. The fixed-point characterization of equilibria is generalized to unobservable mixed strategies. It is also shown that the pure-strategy equilibria consist of elementary subpaths, which are repeating fragments that give the acceptable action plans in the game. The developed methodology offers a novel way of computing and analyzing equilibrium strategies that need not be stationary nor Markovian.


Game theory Stochastic game Subgame-perfect equilibrium Equilibrium path Fixed-point equation Tree 



The author is grateful for the reviewers’ comments and suggestions for improvements. The author acknowledges funding from Emil Aaltosen Säätiö through Post doc -pooli.


  1. 1.
    Abreu D (1986) Extremal equilibria of oligopolistic supergames. J Econ Theory 39(2):191–225MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Abreu D (1988) On the theory of infinitely repeated games with discounting. Econometrica 56(2):383–396MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Abreu D, Pearce D, Stacchetti E (1986) Optimal cartel equilibria with imperfect monitoring. J Econ Theory 39(1):251–269MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Abreu D, Pearce D, Stacchetti E (1990) Toward a theory of discounted repeated games with imperfect monitoring. Econometrica 58(5):1041–1063MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Amir R (2003) Stochastic games in economics and related fields: an overview. In: Neyman A, Sorin S (eds) Stochastic games and applications, NATO Science, vol 570. Kluwer, Dordrecht, pp 455–470CrossRefGoogle Scholar
  6. 6.
    Atkeson A (1991) International lending with moral hazard and risk of repudiation. Econometrica 59(4):1069–1089MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Aumann RJ (1981) Survey of repeated games. Essays in game theory and mathematical economics in Honor of Oskar Morgenstern. Bibliographisches Insitut, Mannheim, pp 11–42Google Scholar
  8. 8.
    Balbus L, Reffett K, Wozny L (2013) Markov stationary equilibria in stochastic supermodular games with imperfect private and public information. Dyn Games Appl 3:187–206MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Balbus L, Reffett K, Wozny L (2014) A constructive study of Markov equilibria in stochastic games with strategic complementarities. J Econ Theory 150:815–840MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Berg K (2013) Extremal pure strategies and monotonicity in repeated games. Working paperGoogle Scholar
  11. 11.
    Berg K, Kärki M (2014) An algorithm for finding the minimal pure-strategy subgame-perfect equilibrium payoffs in repeated games. Working paperGoogle Scholar
  12. 12.
    Berg K, Kitti M (2012) Equilibrium paths in discounted supergames. Working paperGoogle Scholar
  13. 13.
    Berg K, Kitti M (2013) Computing equilibria in discounted \(2\times 2\) supergames. Comput Econ 41(1):71–88CrossRefGoogle Scholar
  14. 14.
    Berg K, Kitti M (2014) Fractal geometry of equilibrium payoffs in discounted supergames. Fractals 22(4). doi: 10.1142/S0218348X14500169
  15. 15.
    Berg K, Schoenmakers G (2014) Construction of randomized subgame-perfect equilibria in repeated games. Working paperGoogle Scholar
  16. 16.
    Blackwell D, Ferguson TS (1968) The big match. Ann Math Stat 39(1):159–163MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Borkovsky RN, Doraszelski U, Kryukov Y (2010) A user’s guide to solving dynamic stochastic games using the homotopy method. Oper Res 58(2):1116–1132MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Burkov A, Chaib-Draa B (2010) An approximate subgame-perfect equilibrium computation technique for repeated games. In: Proceedings of the twenty-fourth AAAI conference on artificial intelligence, pp 729–736Google Scholar
  19. 19.
    Cole HL, Kocherlakota N (2001) Dynamic games with hidden actions and hidden states. J Econ Theory 98:114–126MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Cronshaw MB (1997) Algorithms for finding repeated game equilibria. Comput Econ 10:139–168CrossRefzbMATHGoogle Scholar
  21. 21.
    Doraszelski U, Pakes A (2007) A framework for applied dynamic analysis in IO. In: Armstrong M, Porter R (eds) Handbook of industrial organization 3, pp 1887–1966, North-HollandGoogle Scholar
  22. 22.
    Doraszelski U, Escobar JF (2012) Restricted feedback in long term relationships. J Econ Theory 147(1):142–161MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Ericson R, Pakes A (1995) Markov-perfect industry dynamics: a framework for empirical work. Rev Econ Stud 62(1):53–82CrossRefzbMATHGoogle Scholar
  24. 24.
    Feng Z, Miao J, Peralta-Alva A, Santos MS (2014) Numerical simulation of nonoptimal dynamic equilibrium models. Int Econ Rev 55(1):83–110MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    Filar J, Vrieze K (1997) Competitive markov decision processes. Springer, BerlinzbMATHGoogle Scholar
  26. 26.
    Fink AM (1964) Equilibrium in stochastic n-person games. J Sci Hiroshima Univ Univ Ser A-I 28:89–93MathSciNetzbMATHGoogle Scholar
  27. 27.
    Flesch J, Thuijsman F, Vrieze OJ (1997) Cyclic Markov equilibria in stochastic games. Int J Game Theory 26:303–314MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Flesch J, Thuijsman F, Vrieze OJ (2000) Almost stationary \(\epsilon \)-equilibria in zero-sum stochastic games. J Optim Theory Appl 105(2):371–389MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    Fudenberg D, Yamamoto Y (2011) The folk theorem for irreducible stochastic games with imperfect public monitoring. J Econ Theory 146:1664–1683MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Govindan S, Wilson R (2009) Global Newton method for stochastic games. J Econ Theory 144:414–421MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    Herings JJ, Peeters R (2004) Stationary equilibria in stochastic games: structure, selection and computation. J Econ Theory 118(1):32–60MathSciNetzbMATHGoogle Scholar
  32. 32.
    Hörner J, Sugaya T, Takahashi S, Vieille N (2011) Recursive methods in discounted stochastic games: an algorithm for \(\delta \rightarrow 1\) and a folk theorem. Econometrica 79(4):1277–1318MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Judd K, Yeltekin Ş, Conklin J (2003) Computing supergame equilibria. Econometrica 71:1239–1254MathSciNetCrossRefzbMATHGoogle Scholar
  34. 34.
    Kitti M (2013a) Conditional Markov equilibria in discounted dynamic games. Math Methods Oper Res 78:77–100MathSciNetCrossRefzbMATHGoogle Scholar
  35. 35.
    Kitti M (2013) Subgame perfect equilibria in discounted stochastic games. Working paperGoogle Scholar
  36. 36.
    Levy Y (2013) Discounted stochastic games with no stationary Nash equilibrium: two examples. Econometrica 81(5):1973–2007MathSciNetCrossRefzbMATHGoogle Scholar
  37. 37.
    Levy Y, McLennan A (2015) Corrigendum to: discounted stochastic games with no stationary Nash equilibrium: two examples. Econometrica (to appear)Google Scholar
  38. 38.
    MacDermed L, Narayan KS, Isbell CL, Weiss L (2011) Quick polytope approximation of all correlated equilibria in stochastic games. In: Proceedings of the twenty-fifth AAAI conference on artificial intelligence, pp 707–712Google Scholar
  39. 39.
    Mailath GJ, Samuelson L (2006) Repeated games and reputations: long-run relationships. Oxford University Press, OxfordCrossRefGoogle Scholar
  40. 40.
    Mertens J-F, Parthasarathy TES (1987) Equilibria for discounted stochastic games. CORE discussion paper 8750, Universite Catholique de Louvain, BelgiumGoogle Scholar
  41. 41.
    Mertens J-F, Parthasarathy TES (2003) Equilibria for discounted stochastic games. In: Neyman A, Sorin S (eds) Stochastic games and applications. NATO science series C, mathematical and physical sciences, vol 570. Kluwer, Dordrecht, pp 131–172CrossRefGoogle Scholar
  42. 42.
    Murray C, Gordon GJ (2007) Multi-robot negotiation: approximating the set of subgame perfect equilibria in general-sum stochastic games. Adv Neural Inf Process Syst 19:1001–1008Google Scholar
  43. 43.
    Pakes A, Ericson R (1998) Empirical implications of alternative models of firm dynamics. J Econ Theory 79:1–45CrossRefzbMATHGoogle Scholar
  44. 44.
    Phelan C, Stacchetti E (2001) Sequential equilibria in a Ramsey tax model. Econometrica 69(6):1491–1518MathSciNetCrossRefzbMATHGoogle Scholar
  45. 45.
    Prasad H L, Prashanth LA, Shalabh B (2014) Algorithms for Nash equilibria in general-sum stochastic games. Working paperGoogle Scholar
  46. 46.
    Sen A, Shanno DF (2008) Optimization and dynamical systems algorithms for finding equilibria of stochastic games. Opt Method Softw 23(6):975–993MathSciNetCrossRefzbMATHGoogle Scholar
  47. 47.
    Shapley LS (1953) Stochastic games. Proc Natl Acad Sci USA 39:1095–1100MathSciNetCrossRefzbMATHGoogle Scholar
  48. 48.
    Sleet C, Yeltekin S (2015) On the computation of value correspondences for dynamic games. Dyn Games Appl. doi: 10.1007/s13235-015-0139-1
  49. 49.
    Solan E, Vieille N (2001) Quitting games. Math Oper Res 26(2):265–285MathSciNetCrossRefzbMATHGoogle Scholar
  50. 50.
    Takahashi M (1964) Equilibrium points of stochastic non-cooperative n-person games. J Sci Hiroshima Univ Ser A-I 28:95–99MathSciNetzbMATHGoogle Scholar
  51. 51.
    Thuijsman F, Vrieze OJ (1997) The power of threats in stochastic games. In: Bardi et al. (eds) Stochastic games and numerical methods for dynamic games, Birkhauser, pp 339–353Google Scholar
  52. 52.
    Yamamoto Y (2015) Stochastic games with hidden states. Working paperGoogle Scholar
  53. 53.
    Yeltekin S, Cai Y, Judd KL (2015) Computing equilibria of dynamic games. Working paperGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.Department of Mathematics and Systems AnalysisAalto University School of ScienceAaltoFinland

Personalised recommendations