Dynamic Games and Applications

, Volume 1, Issue 2, pp 253–279 | Cite as

Stochastic Games with Unbounded Payoffs: Applications to Robust Control in Economics



We study a discounted maxmin control problem with general state space. The controller is unsure about his model in the sense that he also considers a class of approximate models as possibly true. The objective is to choose a maxmin strategy that will work under a range of different model specifications. This is done by dynamic programming techniques. Under relatively weak conditions, we show that there is a solution to the optimality equation for the maxmin control problem as well as an optimal strategy for the controller. These results are applied to the theory of optimal growth and the Hansen–Sargent robust control model in macroeconomics. We also study a class of zero-sum discounted stochastic games with unbounded payoffs and simultaneous moves and give a brief overview of recent results on stochastic games with weakly continuous transitions and the limiting average payoffs.


Zero-sum stochastic games Robust control Optimal growth theory Macroeconomic dynamics 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Alvarez F, Stokey N (1998) Dynamic programming with homogeneous functions. J Econ Theory 82:167–189 MathSciNetMATHCrossRefGoogle Scholar
  2. 2.
    Başar T, Bernhard P (1995) H -optimal control and related minimax design problems. Birkhäuser, Boston MATHGoogle Scholar
  3. 3.
    Berge C (1963) Topological spaces. MacMillan, New York MATHGoogle Scholar
  4. 4.
    Bertsekas DP, Shreve SE (1978) Stochastic optimal control: the discrete-time case. Academic Press, New York MATHGoogle Scholar
  5. 5.
    Bhattacharya R, Majumdar M (2007) Random dynamical systems: theory and applications. Cambridge University Press, Cambridge MATHCrossRefGoogle Scholar
  6. 6.
    Blackwell D (1965) Discounted dynamic programming. Ann Math Stat 36:226–235 MathSciNetMATHCrossRefGoogle Scholar
  7. 7.
    Brown LD, Purves R (1973) Measurable selections of extrema. Ann Stat 1:902–912 MathSciNetMATHCrossRefGoogle Scholar
  8. 8.
    Couwenbergh HAM (1980) Stochastic games with metric state spaces. Int J Game Theory 9:25–36 MathSciNetMATHCrossRefGoogle Scholar
  9. 9.
    Durán J (2000) On dynamic programming with unbounded returns. Econom Theory 15:339–352 MathSciNetMATHCrossRefGoogle Scholar
  10. 10.
    Durán J (2003) Discounting long-run average growth in stochastic dynamic programs. Econom Theory 22:395–413 MathSciNetMATHCrossRefGoogle Scholar
  11. 11.
    Fan K (1953) Minimax theorems. Proc Natl Acad Sci USA 39:42–47 MATHCrossRefGoogle Scholar
  12. 12.
    González-Trejo JI, Hernández-Lerma O, Hoyos-Reyes LF (2003) Minimax control of discrete-time stochastic systems. SIAM J Control Optim 41:1626–1659 MATHCrossRefGoogle Scholar
  13. 13.
    Hansen LP, Sargent TJ (2008) Robustness. Princeton University Press, Princeton MATHGoogle Scholar
  14. 14.
    Hansen LP, Sargent TJ (2010) Wanting robustness in macroeconomics. In: Friedman BM, Woodford M (eds) Handbook of monetary economics, vol 3, pp 1097–1157 CrossRefGoogle Scholar
  15. 15.
    Hansen LP, Sargent TJ, Tallarini TD (1999) Robust permanent income and pricing. Rev Econ Stud 66:873–907 MATHCrossRefGoogle Scholar
  16. 16.
    Hernández-Lerma O, Lasserre JB (1996) Discrete-time Markov control processes: basic optimality criteria. Springer, New York Google Scholar
  17. 17.
    Hernández-Lerma O, Lasserre JB (1999) Further topics on discrete-time Markov control processes. Springer, New York MATHGoogle Scholar
  18. 18.
    Himmelberg CJ (1975) Measurable relations. Fundam Math 87:53–72 MathSciNetMATHGoogle Scholar
  19. 19.
    Himmelberg CJ, Van Vleck FS (1975) Multifunctions with values in a space of probability measures. J Math Anal Appl 50:108–112 MathSciNetMATHCrossRefGoogle Scholar
  20. 20.
    Iyengar GN (2005) Robust dynamic programming. Math Oper Res 30:257–280 MathSciNetMATHCrossRefGoogle Scholar
  21. 21.
    Jaśkiewicz A (2009) Zero-sum ergodic semi-Markov games with weakly continuous transition probabilities. J Optim Theory Appl 141:321–348 MathSciNetMATHCrossRefGoogle Scholar
  22. 22.
    Jaśkiewicz A (2010) On a continuous solution to the Bellman–Poisson equation in stochastic games. J Optim Theory Appl 145:451–458 MathSciNetMATHCrossRefGoogle Scholar
  23. 23.
    Jaśkiewicz A, Nowak AS (2006) Zero-sum ergodic stochastic games with Feller transition probabilities. SIAM J Control Optim 45:773–789 MathSciNetMATHCrossRefGoogle Scholar
  24. 24.
    Jaśkiewicz A, Nowak AS (2011) Discounted dynamic programming with unbounded returns: application to economic models. J Math Anal Appl 378:450–462 MATHCrossRefGoogle Scholar
  25. 25.
    Kamihigashi T (2007) Stochastic optimal growth with bounded and unbounded utility and with bounded or unbounded shocks. J Math Econ 43:477–500 MathSciNetMATHCrossRefGoogle Scholar
  26. 26.
    Karatzas I, Sudderth WD (2010) Two characterizations of optimality in dynamic programming. Appl Math Optim 61:421–434 MathSciNetMATHCrossRefGoogle Scholar
  27. 27.
    Kuratowski K, Ryll-Nardzewski C (1965) A general theorem on selectors. Bull Acad Pol Sci (Ser Math) 13:397–403 MathSciNetMATHGoogle Scholar
  28. 28.
    Küenle HU (1986) Stochastische Spiele und Entscheidungsmodelle. BG Teubner, Leipzig MATHGoogle Scholar
  29. 29.
    Küenle HU (2007) On Markov games with average reward criterion and weakly continuous transition probabilities. SIAM J Control Optim 45:2156–2168 MathSciNetMATHCrossRefGoogle Scholar
  30. 30.
    Kurano M (1987) Minimax strategies for average cost stochastic games with an application to inventory models. J Oper Res Soc Jpn 30:232–247 MathSciNetMATHGoogle Scholar
  31. 31.
    Le Van C, Morhaim L (2002) Optimal growth models with bounded or unbounded returns: a unifying approach. J Econ Theory 105:158–187 MATHCrossRefGoogle Scholar
  32. 32.
    Ljungqvist L, Sargent TJ (2000) Recursive macroeconomic theory. MIT Press, Cambridge Google Scholar
  33. 33.
    Maitra A, Parthasarathy T (1970) On stochastic games. J Optim Theory Appl 5:289–300 MathSciNetMATHCrossRefGoogle Scholar
  34. 34.
    Maitra A, Sudderth W (1993) Borel stochastic games with limsup payoffs. Ann Probab 21:861–885 MathSciNetMATHCrossRefGoogle Scholar
  35. 35.
    Maccheroni F, Marinacci M, Rustichini A (2006) Dynamic variational preferences. J Econ Theory 128:4–44 MathSciNetMATHCrossRefGoogle Scholar
  36. 36.
    Martins-da-Rocha VF, Vailakis Y (2010) Existence and uniqueness of fixed-point for local contractions. Econometrica 78:1127–1141 MathSciNetMATHCrossRefGoogle Scholar
  37. 37.
    Matkowski J, Nowak AS (2011) On discounted dynamic programming with unbounded returns. Econom Theory 46:455–474 MATHCrossRefGoogle Scholar
  38. 38.
    Mertens JF, Neyman A (1981) Stochastic games. Int J Game Theory 10:53–66 MathSciNetMATHCrossRefGoogle Scholar
  39. 39.
    Meyn SP, Tweedie RL (1993) Markov chains and stochastic stability. Springer, New York MATHGoogle Scholar
  40. 40.
    Meyn SP, Tweedie RL (1994) Computable bounds for geometric convergence rates of Markov chains. Ann Appl Probab 4:981–1011 MathSciNetMATHCrossRefGoogle Scholar
  41. 41.
    Neveu J (1965) Mathematical foundations of the calculus of probability. Holden-Day, San Francisco MATHGoogle Scholar
  42. 42.
    Nowak AS (1985) Universally measurable strategies in zero-sum stochastic games. Ann Probab 13:269–287 MathSciNetMATHCrossRefGoogle Scholar
  43. 43.
    Nowak AS (1986) Semicontinuous nonstationary stochastic games. J Math Anal Appl 117:84–99 MathSciNetMATHCrossRefGoogle Scholar
  44. 44.
    Nowak AS (2010) On measurable minimax selectors. J Math Anal Appl 366:385–388 MathSciNetMATHCrossRefGoogle Scholar
  45. 45.
    Parthasarathy KR (1967) Probability measures on metric spaces. Academic Press, New York MATHGoogle Scholar
  46. 46.
    Petersen IR, James MR, Dupuis P (2000) Minimax optimal control of stochastic uncertain systems with relative entropy constraints. IEEE Trans Autom Control 45:398–412 MathSciNetMATHCrossRefGoogle Scholar
  47. 47.
    Puterman M (1980) Markov decision processes: discrete stochastic dynamic programming. Wiley-Interscience, New York Google Scholar
  48. 48.
    Rincón-Zapatero JP, Rodriguez-Palmero C (2003) Existence and uniqueness of solutions to the Bellman equation in the unbounded case. Econometrica 71:1519–1555. Corrigendum Econometrica 77:317–318 MathSciNetMATHCrossRefGoogle Scholar
  49. 49.
    Schäl M (1975) Conditions for optimality in dynamic programming and for the limit of n-stage optimal policies to be optimal. Z Wahrscheinlichkeitstheor Verw Geb 32:179–196 MATHCrossRefGoogle Scholar
  50. 50.
    Shapley LS (1953) Stochastic games. Proc Natl Acad Sci USA 39:1095–1100 MathSciNetMATHCrossRefGoogle Scholar
  51. 51.
    Stokey NL, Lucas RE, Prescott E (1989) Recursive methods in economic dynamics. Harvard University Press, Cambridge MATHGoogle Scholar
  52. 52.
    Strauch R (1966) Negative dynamic programming. Ann Math Stat 37:871–890 MathSciNetMATHCrossRefGoogle Scholar
  53. 53.
    Wessels J (1977a) Markov programming by successive approximations with respect to weighted supremum norms. J Math Anal Appl 58:326–335 MathSciNetMATHCrossRefGoogle Scholar
  54. 54.
    Wessels J (1977b) Markov games with unbounded rewards. In: Schäl M (ed) Dynamische Optimierung, Bonn, 1977. Bonner Mathematische Schriften, vol 98, pp 133–147 Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  1. 1.Institute of Mathematics and Computer ScienceWrocław University of TechnologyWrocławPoland
  2. 2.Faculty of Mathematics, Computer Science and EconometricsUniversity of Zielona GóraZielona GóraPoland
  3. 3.Institute of FinancePWSZ NysaPoland

Personalised recommendations