Skip to main content
SpringerLink
Log in

Continuous-Time Markov Decision Processes Under the Risk-Sensitive First Passage Discounted Cost Criterion

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

This paper studies the risk-sensitive first passage discounted cost criterion for continuous-time Markov decision processes with the Borel state and action spaces. The cost and transition rates are allowed to be unbounded. We introduce a new value iteration to establish the existence of a solution to the risk-sensitive first passage discounted cost optimality equation. Then applying the Feynman–Kac formula, we show that the risk-sensitive first passage discounted cost optimal value function is a unique solution to the risk-sensitive first passage discounted cost optimality equation. Moreover, we derive the existence of a deterministic Markov optimal policy in the class of randomized history-dependent policies. Finally, a cash flow model is given to illustrate the results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Aliprantis, C., Border, K.: Infinite Dimensional Analysis. Springer, New York (2006)

    MATH  Google Scholar 

  2. Bäuerle, N., Rieder, U.: More risk-sensitive Markov decision processes. Math. Oper. Res. 39, 105–120 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  3. Bogachev, V.I.: Measure Theory, vol. I. Springer, Berlin (2007)

    Book  MATH  Google Scholar 

  4. Cavazos-Cadena, R.: Characterization of the optimal risk-sensitive average cost in denumerable Markov decision chains. Math. Oper. Res. 43, 1025–1050 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  5. Chandan, P., Somnath, P.: Risk sensitive control of pure jump processes on a general state space. Stochastics 91, 155–174 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  6. Di Masi, G.B., Stettner, Ł: Risk-sensitive control of discrete-time Markov processes with infinite horizon. SIAM J. Control Optim. 38, 61–78 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  7. Di Masi, G.B., Stettner, Ł: Infinite horizon risk sensitive control of discrete time Markov processes under minorization property. SIAM J. Control Optim. 46, 231–252 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  8. Ghosh, M., Saha, S.: Risk-sensitive control of continuous time Markov chains. Stochastics 86, 655–675 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  9. Guo, X., Huang, Y.H.: Risk-sensitive average continuous-time Markov decision processes with unbounded transition and cost rates. J. Appl. Probab. 58, 523–550 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  10. Guo, X., Liu, Q.L., Zhang, Y.: Finite horizon risk-sensitive continuous-time Markov decision processes with unbounded transition and cost rates. 4OR 17, 427–442 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  11. Guo, X.P., Hernández-Lerma, O.: Continuous-Time Markov Decision Processes: Theory and Applications. Springer, Berlin (2009)

    Book  MATH  Google Scholar 

  12. Guo, X.P., Huang, X.X., Zhang, Y.: On the first passage \(g\)-mean-variance optimality for discounted continuous-time Markov decision processes. SIAM J. Control Optim. 53, 1406–1424 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  13. Guo, X.P., Huang, Y.H., Song, X.Y.: Linear programming and constrained average optimality for general continuous-time Markov decision processes in history-dependent policies. SIAM J. Control Optim. 50, 23–47 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  14. Guo, X.P., Song, X.Y.: Discounted continuous-time constrained Markov decision processes in Polish spaces. Ann. Appl. Probab. 21, 2016–2049 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  15. Guo, X.P., Liao, Z.W.: Risk-sensitive discounted continuous-time Markov decision processes with unbounded rates. SIAM J. Control Optim. 57, 3857–3883 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  16. Guo, X.P., Zhang, J.Y.: Risk-sensitive continuous-time Markov decision processes with unbounded rates and Borel spaces. Discrete Event Dyn. Syst. 29, 445–471 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  17. Hernández-Hernández, D., Marcus, S.I., Fard, P.J.: Analysis of a risk-sensitive control problem for hidden Markov chains. IEEE Trans. Autom. Control 44, 1093–1100 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  18. Hernández-Lerma, O., Lasserre, J.B.: Further Topics on Discrete-time Markov Control Processes. Springer, New York (1999)

    Book  MATH  Google Scholar 

  19. Huang, X.X., Liu, Q.L., Guo, X.P.: \(N\)-person nonzero-sum games for continuous-time jump processes with varying discount factors. IEEE Trans. Autom. Control 64, 2037–2044 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  20. Kitaev, M.Y., Rykov, V.V.: Controlled Queueing Systems. CRC Press, Boca Ration (1995)

    MATH  Google Scholar 

  21. Piunovskiy, A., Zhang, Y.: Continuous-Time Markov Decision Processes: Borel Space Models and General Control Strategies. Springer, Cham (2020)

    Book  MATH  Google Scholar 

  22. Shen, Y., Stannat, W., Obermayer, K.: Risk-sensitive Markov control processes. SIAM J. Control Optim. 51, 3652–3672 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  23. Stein, E.M., Shakarchi, R.: Real Analysis: Measure Theory, Integration, and Hilbert Spaces. Princeton University Press, Princeton (2005)

    Book  MATH  Google Scholar 

  24. Suresh Kumar, K., Pal, C.: Risk-sensitive ergodic control of continuous time Markov processes with denumerable state space. Stoch. Anal. Appl. 33, 863–881 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  25. Wei, Q.D.: Continuous-time Markov decision processes with risk-sensitive finite-horizon cost criterion. Math. Methods Oper. Res. 84, 461–487 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  26. Wei, Q.D., Chen, X.: Continuous-time Markov decision processes under the risk-sensitive average cost criterion. Oper. Res. Lett. 44, 457–462 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  27. Wei, Q.D., Chen, X.: Risk-sensitive average continuous-time Markov decision processes with unbounded rates. Optimization 68, 773–800 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  28. Zhang, Y.: Continuous-time Markov decision processes with exponential utility. SIAM J. Control Optim. 55, 2636–2660 (2017)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

We are greatly indebted to the reviewers for the valuable comments and suggestions which have greatly improved the presentation. The research of the first author was supported by the National Natural Science Foundation of China (Grant No. 12171170) and Natural Science Foundation of Fujian Province (Grant No. 2021J01308). The research of the second author was supported by the National Natural Science Foundation of China (Grant No. 12271454).

Author information

Authors and Affiliations

  1. School of Economics and Finance, Huaqiao University, Quanzhou, 362021, People’s Republic of China

    Qingda Wei

  2. School of Mathematical Sciences, Xiamen University, Xiamen, 361005, People’s Republic of China

    Xian Chen

Authors
  1. Qingda Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xian Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xian Chen.

Ethics declarations

Conflict of interest

We declare that no conflict of interest exists in this paper.

Additional information

Communicated by Jörg Rambau.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wei, Q., Chen, X. Continuous-Time Markov Decision Processes Under the Risk-Sensitive First Passage Discounted Cost Criterion. J Optim Theory Appl 197, 309–333 (2023). https://doi.org/10.1007/s10957-023-02179-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-023-02179-3

Keywords

Mathematics Subject Classification

Search

Navigation