Reduction of Discounted Continuous-Time MDPs with Unbounded Jump and Reward Rates to Discrete-Time Total-Reward MDPs
This chapter discusses a reduction of discounted continuous-time Markov decision processes (CTMDPs) to discrete-time Markov decision processes (MDPs). This reduction is based on the equivalence of a randomized policy that chooses actions only at jump epochs to a nonrandomized policy that can switch actions between jumps. For discounted CTMDPs with bounded jump rates, this reduction was introduced by the author in 2004 as a reduction to discounted MDPs. Here we show that this reduction also holds for unbounded jump and reward rates, but the corresponding MDP may not be discounted. However, the analysis of the equivalent total-reward MDP leads to the description of optimal policies for the CTMDP and provides methods for their computation.
KeywordsOptimal Policy Reward Function Jump Rate Reward Rate Total Reward
This research was partially supported by NSF grants CMMI-0900206 and CMMI-0928490.
- 4.Blackwell, D.: Positive dynamic programming. In Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability 1 415–418, University of California Press, Berkeley (1967).Google Scholar
- 15.Hernández-Lerma O.: Lectures on Continuous-Time Markov Control Processes. Aportaciones Mathemáticas. 3. Sociedad Mathemática Mexicana, México (1994)Google Scholar
- 17.Hordijk, A. and F. A. van der Duyn Schouten: Discretization procedures for continuous time decision processes. In Transactions of the Eighth Prague Conference on Information Theory, Statistical Decision Functions, Random Processes, Volume C, pp. 143–154, Academia, Prague (1979)Google Scholar
- 22.Kallianpur, G.:Stochastic Filtering Theory. Springer, New York (1980)Google Scholar
- 27.Miller, B.L.: 1968. Finite state continuous time Markov decision processes with a finite planning horizon. SIAM J. Control 6 266–280 (1968)Google Scholar
- 29.Piunovskiy, A., Zhang, Y.: The transformation method for continuous-time Markov decision processes, Preprint, University of Liverpool, 2011Google Scholar
- 36.Zachrisson, L.E.: Markov games. In: Dresher, M., Shapley, L., Tucker A. (eds.) Advances in Game Theory, pp. 211–253. Princeton University Press, Princeton, N.J. (1964)Google Scholar