Abstract
This paper concerns two-person zero-sum games for a class of average-payoff continuous-time Markov processes in Polish spaces. The underlying processes are determined by transition rates that are allowed to be unbounded, and the payoff function may have neither upper nor lower bounds. We use two optimality inequalities to replace the so-called optimality equation in the previous literature. Under more general conditions, these optimality inequalities yield the existence of the value of the game and of a pair of optimal stationary strategies. Under some additional conditions we further establish the optimality equation itself. Finally, we use several examples to illustrate our results, and also to show the difference between the conditions in this paper and those in the literature. In particular, one of these examples shows that our approach is more general than all of the existing ones because it allows nonergodic Markov processes.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Altman E, Hordijk A, Spieksma F M. Contraction conditions for average and α-discount optimality in countable state Markov games with unbounded rewards. Math Meth Oper Res, 1997, 22: 588–618
Anderson W J. Continuous-Time Markov Chains. New York: Springer-Verlag, 1991
Bayraktar E, Poor V H. Stochastic differential games in a non-Markovian setting. SIAM J Control Optim, 2005, 43: 1737–1756
Buckdahn R, Cardaliaguet P, Rainer C. Nash equilibrium payoffs for nonzero-sum stochastic differential games. SIAM J Control Optim, 2004, 43: 624–642
Borkar V S, Ghosh M K. Denumerable state stochastic games with limiting average payoff. J Optim Theory Appl, 1993, 76: 539–560
Chen M F. From Markov Chains to Non-Equilibrium Particle Systems. 2nd ed. Singapore: World Scientific, 2004
Down D, Meyn S P, Tweedie R L. Exponential and uniform ergodicity of Markov processes. Ann Probab, 1995, 23: 1671–1691
Doshi B T. Continuous-time control of Markov processes on an arbitrary state space: average reward criterion. Stoch Proc Appl, 1976, 4: 55–77
Feller W. On the integro-differential equations of purely discontinuous Markoff processes. Trans Amer Math Soc, 1940, 48: 488–515
Gihman I I, Skorohod A V. Controlled Stochastic Processes. New York: Springer-Verlag, 1979
Guo X P. Continuous-time Markov decision processes with discounted rewards: the case of Polish spaces. Math Oper Res, 2007, 32: 73–87
Guo X P, Hernández-Lerma O. Zero-sum games for continuous-time Markov chains with unbounded transition and average payoff rates. J Appl Probab, 2003, 40: 327–345
Guo X P, Hernández-Lerma O. Zero-sum games for continuous-time jump Markov processes in Polish spaces: discounted payoffs. Adv Appl Probab, 2007, 39: 645–668
Guo X P, Ye L E. New discount and average optimality conditions for continuous-time Markov decision process. Adv Appl Probab, 2010, 42: 953–985
Hamadène S. Mixed zero-sum stochastic differential game and American game options. SIAM J Control Optim, 2006, 45: 496–518
Küenle H U, Schurath R. The optimality equation and ∈-optimal strategies in Markov games with average reward criterion. Math Meth Oper Res, 2002, 56: 451–471
Hernández-Lerma O, Lasserre J B. Zero-sum stochastic games in Borel spaces: average payoff criterion. SIAM J Control Optim, 2001, 39: 1520–1539
Hernández-Lerma O, Lasserre J B. Further Topics on Discrete-Time Markov Control Processes. New York: Springer-Verlag, 1999
Jaśskiewicz A. Zero-sum semi-Markov games. SIAM J Control Optim, 2002, 41: 723–739
Jaśkiewicz A, Nowak A S. Zero-sum ergodic stochastic games with Feller transition probabilities. SIAM J Control Optim, 2006, 45: 73–789
Jaśkiewicz A, Nowak A S. On the optimality equation for zero-sum ergodic stochastic games. Math Meth Oper Res, 2001, 54: 291–301
Lal A K, Sinha S. Zero-sum two-person semi-Markov games. J Appl Probab, 1992, 29: 56–72
Lund R B, Meyn S P, Tweedie R L. Computable exponential convergence rates for stochastically ordered Markov processes. Ann Appl Probab, 1996, 6: 218–237
Nowak A S. Measurable selection theorems for minimax stochastic optimization problems. SIAM J Control Optim, 1985, 23: 466–476
Nowak A S. Optimal strategies in a class of zero-sum ergodic stochastic games. Math Meth Oper Res, 1999, 50: 399–419
Nowak A S. Some remarks on equilibria in semi-Markov games. Appl Math (Warsaw), 2000, 27: 385–394
Orey S. Lecture Notes on Limit Theorems for Markov Chain Transition Probabilities. London: Van Nostrand Reinhold, 1971
Polowczuk W. Nonzero-sum semi-Markov games with countable state spaces. Appl Math (Warsaw), 2000, 27: 395–402
Rao M M. Stochastic Processes: General Theory. Dordrecht: Kluwer Academic Publishers, 1995
Tanaka K, Wakata K. On continuous time Markov games with the expected average reward criterion. Sci Rep Niigata Univ Ser A, 1977, 14: 15–24
Vega-Amaya O. Zero-sum average semi-Markov games: fixed-point solutions of the Shapley equation. SIAM J Control Optim, 2003, 42: 1876–1894
Williams D. Diffusions, Markov Processes, and Martingales. Volume 1: Foundations. Chichester: Wiley, 1979
Yeung D W K. Solution mechanisms for cooperative stochastic differential games. Int Game Theory Rev, 2006, 8: 309–326
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Guo, X., Hernández-Lerma, O. New optimality conditions for average-payoff continuous-time Markov games in Polish spaces. Sci. China Math. 54, 793–816 (2011). https://doi.org/10.1007/s11425-011-4186-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11425-011-4186-9