Applied Mathematics & Optimization

, Volume 78, Issue 3, pp 587–611 | Cite as

Zero-Sum Discounted Reward Criterion Games for Piecewise Deterministic Markov Processes

  • O. L. V. CostaEmail author
  • F. Dufour


This papers deals with the zero-sum game with a discounted reward criterion for piecewise deterministic Markov process (PDMPs) in general Borel spaces. The two players can act on the jump rate and transition measure of the process, with the decisions being taken just after a jump of the process. The goal of this paper is to derive conditions for the existence of min–max strategies for the infinite horizon total expected discounted reward function, which is composed of running and boundary parts. The basic idea is, by using the special features of the PDMPs, to re-write the problem via an embedded discrete-time Markov chain associated to the PDMP and re-formulate the problem as a discrete-stage zero sum game problem.


Zero sum games Continuous-time Discounted reward criterion General borel spaces 

Mathematics Subject Classification

Primary 90C40 Secondary 91A05 91A15 93E03 



The authors would like to thank the anonymous reviewers for their valuable comments and suggestions to improve the paper. This work was partially supported by FAPESP (Research Council of the State of São Paulo) Grant 2013/50759-3. O.L.V. Costa received financial support from CNPq (Brazilian National Research Council), Grant 304091/2014-6, project INCT under the Grant CNPq 465755/2014-3, FAPESP 2014/50851-0, and FAPESP/BG Brasil through the Research Centre for Gas Innovation, FAPESP Grant 2014/50279-4.


  1. 1.
    Costa, O.L.V., Dufour, F.: Continuous average control of piecewise deterministic Markov processes. Springer, New York (2013)CrossRefGoogle Scholar
  2. 2.
    Davis, M.H.A.: Piecewise-deterministic Markov processes: a general class of non-diffusion stochastic models. J. R. Stat. Soc. (B) 46(3), 353–388 (1984)zbMATHGoogle Scholar
  3. 3.
    Davis, M.H.A.: Markov Models and Optimization. Chapman and Hall, London (1993)CrossRefGoogle Scholar
  4. 4.
    Davis, M.H.A., Dempster, M.A.H., Sethi, S.P., Vermes, D.: Optimal capacity expansion under uncertainty. Adv. Appl. Probab. 19(1), 156–176 (1987)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Fan, K.: Minimax theorems. Proc. Natl. Acad. Sci. USA 39, 42–47 (1953)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Filar, J.A., Vrieze, K.: Competitive Markov decision processes. Springer, New York (1997)zbMATHGoogle Scholar
  7. 7.
    Gonzáles-Trejo, J.I., Hernández-Lerma, O., Hoyos-Reyes, L.F.: Minimax control of discrete-time stochastic systems. SIAM J. Control Optim. 41, 1626–1659 (2003)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Guo, X., Hernanez-Lerma, O.: Zero-sum games for continuous-time jump Markov processes in polish spaces: discounted payoffs. Adv. Appl. Probab. 39, 646–668 (2007)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Guo, X.P., Hernández-Lerma, O.: New optimality conditions for average-payoff continuous-time Markov games in Polish spaces. Sci. China Math. 54, 793–816 (2011)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Hernández-Lerma, O., Lasserre, J.B.: Zero-sum stochastic games in Borel spaces: average payoff criterion. SIAM J. Control Optim. 39, 1520–1539 (2001)CrossRefGoogle Scholar
  11. 11.
    Hernández-Lerma, O., Lasserre, J.B.: Further Topics on Discrete-Time Markov Control Processes. Applications of Mathematics, vol. 42. Springer, New York (1999)CrossRefGoogle Scholar
  12. 12.
    Jacod, J.: Calcul stochastique et problèmes de martingales. Lecture Notes in Mathematics, vol. 714. Springer, Berlin (1979)CrossRefGoogle Scholar
  13. 13.
    Jaśkiewicz, A.: Zero-sum semi-Markov games. SIAM J. Control Optim. 41, 723–739 (2002)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Jaśkiewicz, A.: Zero-sum ergodic semi-Markov games with weakly continuous transition probabilities. J. Optim. Theory Appl. 141, 321–347 (2009)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Jaskiewicz, A., Nowak, A.S.: Zero-sum ergodic stochastic games with Feller transition probabilities. SIAM J. Control Optim. 45(3), 773–789 (2006)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Jaśkiewicz, A., Nowak, A.S.: Stochastic games with unbounded payoffs: applications to robust control in economics. Dyn. Games Appl. 1, 253–279 (2011)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Kuenle, Heinz-Uwe: On Markov games with average reward criterion and weakly continuous transition probabilities. SIAM J. Control Optim. 45, 2156–2168 (2007)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Nowak, A.S.: Measurable selection theorems for minimax stochastic optimization problems. SIAM J. Control Optim. 23, 466–476 (1985)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Rieder, U.: On semi-continuous dynamic games. Technical report, University of Karlsruhe, Karlsruhe, Germany, 1978Google Scholar
  20. 20.
    Tweedie, Richard L., Lund, Robert B., Meyn, Sean P.: Computable exponential convergence rates for stochastically ordered markov processes. Ann. Appl. Probab. 6(1), 218–237 (1996)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Van der Duyn Schouten, F.A.: Markov decision drift processes. In: Janssen, J. (ed.) Semi-Markov Models: Theroy and Applications, Chapter 2, pp. 63–78. Springer, New York (1984)Google Scholar
  22. 22.
    Vega-Amaya, O.: Zero-sum average semi-Markov games: fixed-point solutions of the Shapley equation. SIAM J. Control Optim. 42, 1876–1894 (2003)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.Departamento de Engenharia de Telecomunicações e ControleEscola Politécnica da Universidade de São PauloSão PauloBrazil
  2. 2.Institut Polytechnique de BordeauxInstitut de Mathématiques de Bordeaux, INRIA Bordeaux Sud OuestTalence CedexFrance

Personalised recommendations