Skip to main content
SpringerLink
Log in

First Passage Exponential Optimality Problem for Semi-Markov Decision Processes

  • Conference paper
  • First Online:
Modern Trends in Controlled Stochastic Processes:

Part of the book series: Emergence, Complexity and Computation ((ECC,volume 41))

Abstract

This paper deals with the exponential utility maximization problem for semi-Markov decision process with Borel state and action spaces, and nonnegative reward rates. The criterion to be optimized is the expected exponential utility of the total rewards before the system state enters the target set. Under the regular and compactness-continuity conditions, we establish the corresponding optimality equation, and prove the existence of an exponential utility optimal stationary policy by an invariant embedding technique. Moreover, we provide an iterative algorithm for calculating the value function as well as the optimal policies. Finally, we illustrate the computational aspects of an optimal policy with an example.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Baüerle, N., Rieder, U.: Markov Decision Processes with Applications to Finance. Springer, Heidelberg (2011)

    Book  Google Scholar 

  2. Baüerle, N., Rieder, U.: More risk-sensitive Markov decision processes. Math. Oper. Res. 39, 105–120 (2014)

    Article  MathSciNet  Google Scholar 

  3. Cao, X.R.: Semi-Markov decision problems and performance sensitivity analysis. IEEE Trans. Autom. Control 48, 758–769 (2003)

    Article  MathSciNet  Google Scholar 

  4. Cavazos-Cadena, R., Montes-De-Oca, R.: Optimal stationary policies in risk-sensitive dynamic programs with finite state space and nonnegative rewards. Appl. Math. (Warsaw) 27, 167–185 (2000)

    Article  MathSciNet  Google Scholar 

  5. Cavazos-Cadena, R., Montes-De-Oca, R.: Nearly optimal policies in risk-sensitive positive dynamic programming on discrete spaces. Math. Meth. Oper. Res. 52, 133–167 (2000)

    Article  MathSciNet  Google Scholar 

  6. Chung, K.J., Sobel, M.J.: Discounted MDP’s: distribution functions and exponential utility maximization. SIAM J. Control Optim. 25, 49–62 (1987)

    Article  MathSciNet  Google Scholar 

  7. Ghosh, M.K., Saha, S.: Risk-sensitive control of continuous time Markov chains. Stochastics 86, 655–675 (2014)

    Article  MathSciNet  Google Scholar 

  8. Ghosh, M.K., Saha, S.: Non-stationary semi-Markov decision processes on a finite horizon. Stoch. Anal. Appl. 31, 183–190 (2013)

    Article  MathSciNet  Google Scholar 

  9. Guo, X., Liu, Q.L., Zhang, Y.: Finite horizon risk-sensitive continuous-time Markov decision processes with unbounded transition and cost rates. 4OR 17, 427–442 (2019)

    Google Scholar 

  10. Guo, X.P., Hernández-Lerma, O.: Continuous-Time Markov Decision Processes: Theory and Applications. Springer, Berlin (2009)

    Book  Google Scholar 

  11. Hernández-Lerma, O., Lasserre, J.B.: Discrete-Time Markov Control Processes: Basic Optimality Criteria. Springer, New York (1996)

    Book  Google Scholar 

  12. Howard, R.A., Matheson, J.E.: Risk-sensitive Markov decision processes. Manage. Sci. 18, 356–369 (1972)

    Article  MathSciNet  Google Scholar 

  13. Huang, Y.H., Guo, X.P.: Discounted semi-Markov decision processes with nonnegative costs. Acta Math. Sin. (Chinese Ser.) 53, 503–514 (2010)

    MathSciNet  MATH  Google Scholar 

  14. Huang, Y.H., Guo, X.P.: Finite horizon semi-Markov decision processes with application to maintenance systems. Eur. J. Oper. Res. 212, 131–140 (2011)

    Article  MathSciNet  Google Scholar 

  15. Huang, Y.H., Guo, X.P.: Mean-variance problems for finite horizon semi-Markov decision processes. Appl. Math. Optim. 72, 233–259 (2015)

    Article  MathSciNet  Google Scholar 

  16. Huang, Y.H., Guo, X.P., Song, X.Y.: Performance analysis for controlled semi-Markov process. J. Optim. Theory Appl. 150, 395–415 (2011)

    Article  MathSciNet  Google Scholar 

  17. Huang, Y.H., Lian, Z.T., Guo, X.P.: Risk-sensitive semi-Markov decision processes with general utilities and multiple criteria. Adv. Appl. Probab. 50, 783–804 (2018)

    Article  MathSciNet  Google Scholar 

  18. Huang, X.X., Zou, X.L., Guo, X.P.: A minimization problem of the risk probability in first passage semi-Markov decision processes with loss rates. Sci. China Math. 58, 1923–1938 (2015)

    Article  MathSciNet  Google Scholar 

  19. Huo, H.F., Zou, X.L., Guo, X.P.: The risk probability criterion for discounted continuous-time Markov decision processes. Discrete Event Dyn. Syst. 27, 675–699 (2017)

    Article  MathSciNet  Google Scholar 

  20. Janssen, J., Manca, R.: Semi-Markov Risk Models for Finance, Insurance, and Reliability. Springer, New York (2006)

    MATH  Google Scholar 

  21. Jaquette, S.C.: A utility criterion for Markov decision processes. Manage. Sci. 23, 43–49 (1976)

    Article  MathSciNet  Google Scholar 

  22. Jaśkiewicz, A.: A note on negative dynamic programming for risk-sensitive control. Oper. Res. Lett. 36, 531–534 (2008)

    Article  MathSciNet  Google Scholar 

  23. Jaśkiewicz, A.: On the equivalence of two expected average cost criteria for semi Markov control processes. Math. Oper. Res. 29, 326–338 (2013)

    Article  MathSciNet  Google Scholar 

  24. Limnios, N., Oprisan, G.: Semi-Markov Processes and Reliability. Birkhäuser, Boston (2001)

    Book  Google Scholar 

  25. Luque-Vásquez, F., Minjárez-Sosa, J.A.: Semi-Markov control processes with unknown holding times distribution under a discounted criterion. Math. Meth. Oper. Res. 61, 455–468 (2005)

    Article  MathSciNet  Google Scholar 

  26. Mamer, J.W.: Successive approximations for finite horizon semi-Markov decision processes with application to asset liquidation. Oper. Res. 34, 638–644 (1986)

    Article  MathSciNet  Google Scholar 

  27. Nollau, V.: Solution of a discounted semi-Markovian decision problem by successive overrelaxation. Optimization 39, 85–97 (1997)

    Article  MathSciNet  Google Scholar 

  28. Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, New York (1994)

    Book  Google Scholar 

  29. Schäl, M.: Control of ruin probabilities by discrete-time investments. Math. Meth. Oper. Res. 70, 141–158 (2005)

    Article  MathSciNet  Google Scholar 

  30. Wei, Q.D.: Continuous-time Markov decision processes with risk-sensitive finite-horizon cost criterion. Math. Meth. Oper. Res. 84, 1–27 (2016)

    Article  MathSciNet  Google Scholar 

  31. Wei, Q.D., Guo, X.P.: New average optimality conditions for semi-Markov decision processes in Borel spaces. J. Optim. Theory Appl. 153, 709–732 (2012)

    Article  MathSciNet  Google Scholar 

  32. Wei, Q.D., Guo, X.P.: Constrained semi-Markov decision processes with ratio and time expected average criteria in Polish spaces. Optimization 64, 1593–1623 (2015)

    Article  MathSciNet  Google Scholar 

  33. Yushkevich, A.A.: On semi-Markov controlled models with average reward criterion. Theory Probab. Appl. 26, 808–815 (1982)

    Article  MathSciNet  Google Scholar 

  34. Zhang, Y.: Continuous-time Markov decision processes with exponential utility. SIAM J. Control Optim. 55, 1–24 (2017)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgement

This work was supported by National Natural Science Foundation of China (Grant No. 11961005, 11801590); Foundation of Guangxi Educational Committee (Grant No. KY2019YB0369); Ph.D. research startup foundation of Guangxi University of Science and Technology (Grant No. 18Z06); Guangxi Natural Science Foundation Program (Grant No. 2020GXNSFAA297196).

Author information

Authors and Affiliations

  1. Department of School of Science, Guangxi University of Science and Technology, Liuzhou, 5451006, China

    Haifeng Huo & Xian Wen

Authors
  1. Haifeng Huo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xian Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Haifeng Huo .

Editor information

Editors and Affiliations

  1. Department of Mathematical Sciences, University of Liverpool, Liverpool, UK

    Alexey Piunovskiy

  2. Department of Mathematical Sciences, University of Liverpool, Liverpool, UK

    Yi Zhang

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Huo, H., Wen, X. (2021). First Passage Exponential Optimality Problem for Semi-Markov Decision Processes. In: Piunovskiy, A., Zhang, Y. (eds) Modern Trends in Controlled Stochastic Processes:. Emergence, Complexity and Computation, vol 41. Springer, Cham. https://doi.org/10.1007/978-3-030-76928-4_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-76928-4_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-76927-7

  • Online ISBN: 978-3-030-76928-4

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation