Finite horizon risk-sensitive continuous-time Markov decision processes with unbounded transition and cost rates

Abstract

We consider a risk-sensitive continuous-time Markov decision process over a finite time duration. Under the conditions that can be satisfied by unbounded transition and cost rates, we show the existence of an optimal policy, and the existence and uniqueness of the solution to the optimality equation out of a class of possibly unbounded functions, to which the Feynman–Kac formula was also justified to hold.

This is a preview of subscription content, log in to check access.

References

  1. Bäuerle N, Rieder U (2014) More risk-sensitive Markov decision processes. Math Oper Res 39:105–120

    Article  Google Scholar 

  2. Bäuerle N, Popp A (2018) Risk-sensitive stopping problems for continuous-time Markov chains. Stochastics 90:411–431

    Article  Google Scholar 

  3. Cavazos-Cadena R, Montes-de-Oca R (2000) Optimal stationary policies in risk-sensitive dynamic programs with finite state space and nonnegative rewards. Appl Math 27:167–185

    Google Scholar 

  4. Cavazos-Cadena R, Montes-de-Oca R (2000) Nearly optimal policies in risk-sensitive positive dynamic programming on discrete spaces. Math Methods Oper Res 52:133–167

    Article  Google Scholar 

  5. Ghosh M, Saha S (2014) Risk-sensitive control of continuous time Markov chains. Stochastics 86:655–675

    Article  Google Scholar 

  6. Guo X, Zhang Y (2018) On risk-sensitive piecewise deterministic Markov decision processes. Appl. Math Optim. in press. https://doi.org/10.1007/s00245-018-9485-x

  7. Guo XP, Huang X, Huang Y (2015) Finite-horizon optimality for continuous-time Markov decision processes with unbounded transition rates. Adv Appl Probab 47:1064–1087

    Article  Google Scholar 

  8. Guo XP, Piunovskiy A (2011) Discounted continuous-time Markov decision processes with constraints: unbounded transition and loss rates. Math Oper Res 36:105–132

    Article  Google Scholar 

  9. Hernández-Lerma O, Lasserre J (1996) Discrete-time Markov control processes. Springer, New York

    Google Scholar 

  10. Hernández-Lerma O, Lasserre J (1999) Further topics on discrete-time Markov control processes. Springer, New York

    Google Scholar 

  11. Howard R, Matheson J (1972) Risk-sensitive Markov decision proceses. Manag Sci 18:356–369

    Article  Google Scholar 

  12. Jacod J (1975) Multivariate point processes: predictable projection, Radon–Nicodym derivatives, representation of martingales. Z. Wahrscheinlichkeitstheorie und verwandte Gebiete 31:235–253

    Article  Google Scholar 

  13. Jaśkiewicz A (2008) A note on negative dynamic programming for risk-sensitive control. Oper Res Lett 36:531–534

    Article  Google Scholar 

  14. Kitaev M (1986) Semi-Markov and jump Markov controlled models: average cost criterion. Theory Probab Appl 30:272–288

    Article  Google Scholar 

  15. Kitaev M, Rykov V (1995) Controlled queueing systems. CRC Press, New York

    Google Scholar 

  16. Kumar KS, Chandan P (2013) Risk-sensitive control of jump process on denumerable state space with near monotone cost. Appl Math Optim 68:311–331

    Article  Google Scholar 

  17. Patek S (2001) On terminating Markov decision processes with a risk-averse objective function. Automatica 37:1379–1386

    Article  Google Scholar 

  18. Piunovski A, Khametov V (1985) New effective solutions of optimality equations for the controlled Markov chains with continuous parameter (the unbounded price-function). Problems Control Inform Theory 14:303–318

    Google Scholar 

  19. Piunovskiy A, Zhang Y (2011) Discounted continuous-time Markov decision processes with unbounded rates: the convex analytic approach. SIAM J Control Optim 49:2032–2061

    Article  Google Scholar 

  20. Piunovskiy A, Zhang Y (2014) Discounted continuous-time Markov decision processes with unbounded rates and randomized history-dependent policies: the dynamic programming approach. 4OR-Q J Oper Res 12, 4975

    Article  Google Scholar 

  21. Wei Q (2016) Continuous-time Markov decision processes with risk-sensitive finite-horizon cost criterion. Math Methods Oper Res 84:461–487

    Article  Google Scholar 

  22. Wei Q, Chen X (2016) Continuous-time Markov decision processes under the risk-sensitive average cost criterion. Oper Res Lett 44:457–462

    Article  Google Scholar 

  23. Zhang Y (2017) Continuous-time Markov decision processes with exponential utility. SIAM J Control Optim 55:2636–2660

    Article  Google Scholar 

Download references

Acknowledgements

This work is partially supported by Natural Science Foundation of Guangdong Province (Grant No. 2014A030313438), Zhujiang New Star (Grant No. 201506010056), Guangdong Province outstanding young teacher training plan (Grant No. YQ2015050).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Yi Zhang.

Ethics declarations

Conflict of interest

There is no potential conflicts of interest.

Ethical standard

Research do not have human participants and/or animals.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Guo, X., Liu, Q. & Zhang, Y. Finite horizon risk-sensitive continuous-time Markov decision processes with unbounded transition and cost rates. 4OR-Q J Oper Res 17, 427–442 (2019). https://doi.org/10.1007/s10288-019-0398-6

Download citation

Keywords

  • Continuous-time Markov decision processes
  • Risk-sensitive criterion
  • Optimality equation

Mathematics Subject Classification

  • Primary 90C40
  • Secondary 60J75