Continuous-Time Controlled Jump Markov Processes on the Finite Horizon

  • Mrinal K. GhoshEmail author
  • Subhamay Saha
Part of the Systems & Control: Foundations & Applications book series (SCFA)


We study continuous-time controlled Markov chains on the finite horizon. For the Markov decision problem, we show that the value function is the unique solution of the corresponding dynamic programming equation. This leads to the existence of an optimal Markov control. We then consider a zero-sum game. We show that the value function exists and is the unique solution of the corresponding Isaacs equations. This yields the existence of a pair of saddle point Markov strategies.


  1. 1.
    A. Arapostathis, V. S. Borkar and M. K. Ghosh, Ergodic Control of Diffusion Processes, Cambridge University Press, 2011.Google Scholar
  2. 2.
    V. E. Benes, Existence of optimal strategies based on specified information for a class of stochastic decision problems, SIAM J. Control 8 (1970), 179–188.MathSciNetzbMATHCrossRefGoogle Scholar
  3. 3.
    M. H. A. Davis, Markov Models and Optimization, Chapman and Hall, 1993.Google Scholar
  4. 4.
    K. Fan, Fixed-point and minimax theorems in locally convex topological linear spaces, Proc. of the Natl. Academy of Sciences of the United States of America 38 (1952), 121–126.zbMATHCrossRefGoogle Scholar
  5. 5.
    X. Guo and O. Hernández-Lerma, Continuous-Time Markov Decision Processes. Theory and Applications, Springer-Verlag, 2009.Google Scholar
  6. 6.
    X. Guo and O. Hernández-Lerma, Zero-sum games for continuous-time jump Markov processes in Polish spaces: discounted payoffs, Adv. in Appl. Probab. 39 (2007), 645–668.MathSciNetzbMATHCrossRefGoogle Scholar
  7. 7.
    S. R. Pliska, Controlled jump processes, Stochastic Processes Appl. 3 (1975), 259–282.MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  1. 1.Department of MathematicsIndian Institute of ScienceBangaloreIndia

Personalised recommendations