Dynamic Games and Applications

, Volume 1, Issue 2, pp 253–279 | Cite as

Stochastic Games with Unbounded Payoffs: Applications to Robust Control in Economics

  • Anna Jaśkiewicz
  • Andrzej S. NowakEmail author


We study a discounted maxmin control problem with general state space. The controller is unsure about his model in the sense that he also considers a class of approximate models as possibly true. The objective is to choose a maxmin strategy that will work under a range of different model specifications. This is done by dynamic programming techniques. Under relatively weak conditions, we show that there is a solution to the optimality equation for the maxmin control problem as well as an optimal strategy for the controller. These results are applied to the theory of optimal growth and the Hansen–Sargent robust control model in macroeconomics. We also study a class of zero-sum discounted stochastic games with unbounded payoffs and simultaneous moves and give a brief overview of recent results on stochastic games with weakly continuous transitions and the limiting average payoffs.


Zero-sum stochastic games Robust control Optimal growth theory Macroeconomic dynamics 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Alvarez F, Stokey N (1998) Dynamic programming with homogeneous functions. J Econ Theory 82:167–189 MathSciNetzbMATHCrossRefGoogle Scholar
  2. 2.
    Başar T, Bernhard P (1995) H -optimal control and related minimax design problems. Birkhäuser, Boston zbMATHGoogle Scholar
  3. 3.
    Berge C (1963) Topological spaces. MacMillan, New York zbMATHGoogle Scholar
  4. 4.
    Bertsekas DP, Shreve SE (1978) Stochastic optimal control: the discrete-time case. Academic Press, New York zbMATHGoogle Scholar
  5. 5.
    Bhattacharya R, Majumdar M (2007) Random dynamical systems: theory and applications. Cambridge University Press, Cambridge zbMATHCrossRefGoogle Scholar
  6. 6.
    Blackwell D (1965) Discounted dynamic programming. Ann Math Stat 36:226–235 MathSciNetzbMATHCrossRefGoogle Scholar
  7. 7.
    Brown LD, Purves R (1973) Measurable selections of extrema. Ann Stat 1:902–912 MathSciNetzbMATHCrossRefGoogle Scholar
  8. 8.
    Couwenbergh HAM (1980) Stochastic games with metric state spaces. Int J Game Theory 9:25–36 MathSciNetzbMATHCrossRefGoogle Scholar
  9. 9.
    Durán J (2000) On dynamic programming with unbounded returns. Econom Theory 15:339–352 MathSciNetzbMATHCrossRefGoogle Scholar
  10. 10.
    Durán J (2003) Discounting long-run average growth in stochastic dynamic programs. Econom Theory 22:395–413 MathSciNetzbMATHCrossRefGoogle Scholar
  11. 11.
    Fan K (1953) Minimax theorems. Proc Natl Acad Sci USA 39:42–47 zbMATHCrossRefGoogle Scholar
  12. 12.
    González-Trejo JI, Hernández-Lerma O, Hoyos-Reyes LF (2003) Minimax control of discrete-time stochastic systems. SIAM J Control Optim 41:1626–1659 zbMATHCrossRefGoogle Scholar
  13. 13.
    Hansen LP, Sargent TJ (2008) Robustness. Princeton University Press, Princeton zbMATHGoogle Scholar
  14. 14.
    Hansen LP, Sargent TJ (2010) Wanting robustness in macroeconomics. In: Friedman BM, Woodford M (eds) Handbook of monetary economics, vol 3, pp 1097–1157 CrossRefGoogle Scholar
  15. 15.
    Hansen LP, Sargent TJ, Tallarini TD (1999) Robust permanent income and pricing. Rev Econ Stud 66:873–907 zbMATHCrossRefGoogle Scholar
  16. 16.
    Hernández-Lerma O, Lasserre JB (1996) Discrete-time Markov control processes: basic optimality criteria. Springer, New York Google Scholar
  17. 17.
    Hernández-Lerma O, Lasserre JB (1999) Further topics on discrete-time Markov control processes. Springer, New York zbMATHGoogle Scholar
  18. 18.
    Himmelberg CJ (1975) Measurable relations. Fundam Math 87:53–72 MathSciNetzbMATHGoogle Scholar
  19. 19.
    Himmelberg CJ, Van Vleck FS (1975) Multifunctions with values in a space of probability measures. J Math Anal Appl 50:108–112 MathSciNetzbMATHCrossRefGoogle Scholar
  20. 20.
    Iyengar GN (2005) Robust dynamic programming. Math Oper Res 30:257–280 MathSciNetzbMATHCrossRefGoogle Scholar
  21. 21.
    Jaśkiewicz A (2009) Zero-sum ergodic semi-Markov games with weakly continuous transition probabilities. J Optim Theory Appl 141:321–348 MathSciNetzbMATHCrossRefGoogle Scholar
  22. 22.
    Jaśkiewicz A (2010) On a continuous solution to the Bellman–Poisson equation in stochastic games. J Optim Theory Appl 145:451–458 MathSciNetzbMATHCrossRefGoogle Scholar
  23. 23.
    Jaśkiewicz A, Nowak AS (2006) Zero-sum ergodic stochastic games with Feller transition probabilities. SIAM J Control Optim 45:773–789 MathSciNetzbMATHCrossRefGoogle Scholar
  24. 24.
    Jaśkiewicz A, Nowak AS (2011) Discounted dynamic programming with unbounded returns: application to economic models. J Math Anal Appl 378:450–462 zbMATHCrossRefGoogle Scholar
  25. 25.
    Kamihigashi T (2007) Stochastic optimal growth with bounded and unbounded utility and with bounded or unbounded shocks. J Math Econ 43:477–500 MathSciNetzbMATHCrossRefGoogle Scholar
  26. 26.
    Karatzas I, Sudderth WD (2010) Two characterizations of optimality in dynamic programming. Appl Math Optim 61:421–434 MathSciNetzbMATHCrossRefGoogle Scholar
  27. 27.
    Kuratowski K, Ryll-Nardzewski C (1965) A general theorem on selectors. Bull Acad Pol Sci (Ser Math) 13:397–403 MathSciNetzbMATHGoogle Scholar
  28. 28.
    Küenle HU (1986) Stochastische Spiele und Entscheidungsmodelle. BG Teubner, Leipzig zbMATHGoogle Scholar
  29. 29.
    Küenle HU (2007) On Markov games with average reward criterion and weakly continuous transition probabilities. SIAM J Control Optim 45:2156–2168 MathSciNetzbMATHCrossRefGoogle Scholar
  30. 30.
    Kurano M (1987) Minimax strategies for average cost stochastic games with an application to inventory models. J Oper Res Soc Jpn 30:232–247 MathSciNetzbMATHGoogle Scholar
  31. 31.
    Le Van C, Morhaim L (2002) Optimal growth models with bounded or unbounded returns: a unifying approach. J Econ Theory 105:158–187 zbMATHCrossRefGoogle Scholar
  32. 32.
    Ljungqvist L, Sargent TJ (2000) Recursive macroeconomic theory. MIT Press, Cambridge Google Scholar
  33. 33.
    Maitra A, Parthasarathy T (1970) On stochastic games. J Optim Theory Appl 5:289–300 MathSciNetzbMATHCrossRefGoogle Scholar
  34. 34.
    Maitra A, Sudderth W (1993) Borel stochastic games with limsup payoffs. Ann Probab 21:861–885 MathSciNetzbMATHCrossRefGoogle Scholar
  35. 35.
    Maccheroni F, Marinacci M, Rustichini A (2006) Dynamic variational preferences. J Econ Theory 128:4–44 MathSciNetzbMATHCrossRefGoogle Scholar
  36. 36.
    Martins-da-Rocha VF, Vailakis Y (2010) Existence and uniqueness of fixed-point for local contractions. Econometrica 78:1127–1141 MathSciNetzbMATHCrossRefGoogle Scholar
  37. 37.
    Matkowski J, Nowak AS (2011) On discounted dynamic programming with unbounded returns. Econom Theory 46:455–474 zbMATHCrossRefGoogle Scholar
  38. 38.
    Mertens JF, Neyman A (1981) Stochastic games. Int J Game Theory 10:53–66 MathSciNetzbMATHCrossRefGoogle Scholar
  39. 39.
    Meyn SP, Tweedie RL (1993) Markov chains and stochastic stability. Springer, New York zbMATHGoogle Scholar
  40. 40.
    Meyn SP, Tweedie RL (1994) Computable bounds for geometric convergence rates of Markov chains. Ann Appl Probab 4:981–1011 MathSciNetzbMATHCrossRefGoogle Scholar
  41. 41.
    Neveu J (1965) Mathematical foundations of the calculus of probability. Holden-Day, San Francisco zbMATHGoogle Scholar
  42. 42.
    Nowak AS (1985) Universally measurable strategies in zero-sum stochastic games. Ann Probab 13:269–287 MathSciNetzbMATHCrossRefGoogle Scholar
  43. 43.
    Nowak AS (1986) Semicontinuous nonstationary stochastic games. J Math Anal Appl 117:84–99 MathSciNetzbMATHCrossRefGoogle Scholar
  44. 44.
    Nowak AS (2010) On measurable minimax selectors. J Math Anal Appl 366:385–388 MathSciNetzbMATHCrossRefGoogle Scholar
  45. 45.
    Parthasarathy KR (1967) Probability measures on metric spaces. Academic Press, New York zbMATHGoogle Scholar
  46. 46.
    Petersen IR, James MR, Dupuis P (2000) Minimax optimal control of stochastic uncertain systems with relative entropy constraints. IEEE Trans Autom Control 45:398–412 MathSciNetzbMATHCrossRefGoogle Scholar
  47. 47.
    Puterman M (1980) Markov decision processes: discrete stochastic dynamic programming. Wiley-Interscience, New York Google Scholar
  48. 48.
    Rincón-Zapatero JP, Rodriguez-Palmero C (2003) Existence and uniqueness of solutions to the Bellman equation in the unbounded case. Econometrica 71:1519–1555. Corrigendum Econometrica 77:317–318 MathSciNetzbMATHCrossRefGoogle Scholar
  49. 49.
    Schäl M (1975) Conditions for optimality in dynamic programming and for the limit of n-stage optimal policies to be optimal. Z Wahrscheinlichkeitstheor Verw Geb 32:179–196 zbMATHCrossRefGoogle Scholar
  50. 50.
    Shapley LS (1953) Stochastic games. Proc Natl Acad Sci USA 39:1095–1100 MathSciNetzbMATHCrossRefGoogle Scholar
  51. 51.
    Stokey NL, Lucas RE, Prescott E (1989) Recursive methods in economic dynamics. Harvard University Press, Cambridge zbMATHGoogle Scholar
  52. 52.
    Strauch R (1966) Negative dynamic programming. Ann Math Stat 37:871–890 MathSciNetzbMATHCrossRefGoogle Scholar
  53. 53.
    Wessels J (1977a) Markov programming by successive approximations with respect to weighted supremum norms. J Math Anal Appl 58:326–335 MathSciNetzbMATHCrossRefGoogle Scholar
  54. 54.
    Wessels J (1977b) Markov games with unbounded rewards. In: Schäl M (ed) Dynamische Optimierung, Bonn, 1977. Bonner Mathematische Schriften, vol 98, pp 133–147 Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  1. 1.Institute of Mathematics and Computer ScienceWrocław University of TechnologyWrocławPoland
  2. 2.Faculty of Mathematics, Computer Science and EconometricsUniversity of Zielona GóraZielona GóraPoland
  3. 3.Institute of FinancePWSZ NysaPoland

Personalised recommendations