Skip to main content
SpringerLink
Log in

A Discount Vanishing Approximation for Markov Decision Processes with Risk Sensitivity

  • Published:
Journal of Dynamical and Control Systems Aims and scope Submit manuscript

Abstract

In this paper optimal control of risk-sensitive Markov decision processes with countable states is studied. The state space is not assumed to be communicating. The focus is on dependence of the optimal values on the transition characteristics-communication, transience or absorption. A vanishing discount approach is used to introduce a partition of the state space, and certain transformation of the optimal values under discount is shown to convergence to the optimal values under risk sensitivity, as the discount factor tends to vanish. The partition of the state space turns out to be closely related to the characteristics of state communication, but weights more on the values under discount.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Data Availability

Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.

References

  1. Alanis-Duran A, Cavazos-Cadena R. An optimality system for finite average Markov decision chains under risk-aversion. Kybernetika(Prague). 2012;48(1):83–104.

  2. Arapostathis A, Borkar VS, Fernandez-Gaucherand E, Ghosh MK, Marcus SI. Discrete-time controlled Markov processes with average cost criterion: a survey. SIAM J Control Optim. 1993;31(2):282–344.

    Article  MathSciNet  Google Scholar 

  3. Bäuerle N, Rieder U. More risk-sensitive Markov decision processes. Math Oper Res. 2014;39(1):105–20.

    Article  MathSciNet  Google Scholar 

  4. Blancas-Rivera R, Cavazos-Cadena R, Cruz-Suarez H. Discounted approximations in risk-sensitive average Markov cost chains with finite state space. Math Methods Oper Res. 2020;91(2):241–68.

    Article  MathSciNet  Google Scholar 

  5. Borkar VS, Meyn SP. Risk-sensitive optimal control for Markov decision processes with monotone cost. Math Oper Res. 2002;27(1):192–209.

    Article  MathSciNet  Google Scholar 

  6. Cavazos-Cadena R, Cruz-Suarez D. Discounted approximations to the risk-sensitive average cost in finite Markov chains. J Math Anal Appl. 2017;450(2):1345–62.

    Article  MathSciNet  Google Scholar 

  7. Cavazos-Cadena R, Fernandez-Gaucherand E. The vanishing discount approach in Markov chains with risk-sensitive criteria. IEEE Trans Automat Control. 2000;45(10):1800–16.

    Article  MathSciNet  Google Scholar 

  8. Cavazos-Cadena R. Characterization of the optimal risk-sensitive average cost in denumerable Markov decision chains. Math Oper Res. 2018;43(3):1025–50.

    Article  MathSciNet  Google Scholar 

  9. Cavazos-Cadena R, Hernandez-Hernandez D. Contractive approximations for the Varadhan’s function on a finite Markov chain. Theory Probab Appl. 2008;52(2):315–23.

    Article  MathSciNet  Google Scholar 

  10. Cavazos-Cadena R, Hernandez-Hernandez D. Discounted approximations for risk-sensitive average criteria in Markov decision chains with finite state space. Math Oper Res. 2011;36(1):133–46.

    Article  MathSciNet  Google Scholar 

  11. Cavazos-Cadena R, Hernandez-Hernandez D. Vanishing discount approximations in controlled Markov chains with risk-sensitive average criterion. Adv Appl Probab. 2018;50(1):204–30.

    Article  MathSciNet  Google Scholar 

  12. Cavazos-Cadena R. Solution to the risk-sensitive average cost optimality equation in a class of Markov decision processes with finite state space. Math Methods Oper Res. 2003;57(2):263–85.

    Article  MathSciNet  Google Scholar 

  13. Cavazos-Cadena R. Solutions of the average cost optimality equation for finite Markov decision chains: risk-sensitive and risk-neutral criteria. Math Methods Oper Res. 2009;70(3):541–66.

    Article  MathSciNet  Google Scholar 

  14. Cavazos-Cadena R, Hernandez-Hernandez D. A system of Poisson equations for a nonconstant Varadhan functional on a finite state space. Appl Math Optim. 2006;53(1):101–19.

    Article  MathSciNet  Google Scholar 

  15. Cavazos-Cadena R, Salem-Silva F. The discounted method and equivalence of average criteria for risk-sensitive Markov decision processes on Borel spaces. Appl Math Optim. 2010;61(2):167–90.

    Article  MathSciNet  Google Scholar 

  16. Di Masi GB, Stettner L. Infinite horizon risk sensitive control of discrete time Markov processes with small risk. Systems Control Lett. 2000;40(1):15–20.

    Article  MathSciNet  Google Scholar 

  17. Howard RA, Matheson JE. Risk-sensitive Markov decision processes. Manag Sci. 1972;18:356–69.

    Article  MathSciNet  Google Scholar 

  18. Jaśkiewicz A. Average optimality for risk-sensitive control with general state space. Ann Appl Probab. 2007;17(2):654–75.

    Article  MathSciNet  Google Scholar 

  19. Di Masi GB, Stettner L. Risk-sensitive control of discrete-time Markov processes with infinite horizon. SIAM J Control Optim. 1999;38(1):61–78.

    Article  MathSciNet  Google Scholar 

  20. Puterman ML. Markov decision processes. Handbooks Oper Res Manag Sci. 1990;2:331–434.

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

The authors are grateful to the referee for valuable comments and suggestions for improvement.

Funding

This work is supported by the NSFC 11671226.

Author information

Authors and Affiliations

  1. Department of Mathematics, Tsinghua University, Beijing, China

    Tanhao Huang, Xiaoyang Lu & Jinwen Chen

Authors
  1. Tanhao Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoyang Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jinwen Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinwen Chen.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, T., Lu, X. & Chen, J. A Discount Vanishing Approximation for Markov Decision Processes with Risk Sensitivity. J Dyn Control Syst (2024). https://doi.org/10.1007/s10883-024-09691-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10883-024-09691-3

Keywords

Mathematics Subject Classification (2010)

Search

Navigation