Abstract
This paper deals with the exponential utility maximization problem for semi-Markov decision process with Borel state and action spaces, and nonnegative reward rates. The criterion to be optimized is the expected exponential utility of the total rewards before the system state enters the target set. Under the regular and compactness-continuity conditions, we establish the corresponding optimality equation, and prove the existence of an exponential utility optimal stationary policy by an invariant embedding technique. Moreover, we provide an iterative algorithm for calculating the value function as well as the optimal policies. Finally, we illustrate the computational aspects of an optimal policy with an example.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Baüerle, N., Rieder, U.: Markov Decision Processes with Applications to Finance. Springer, Heidelberg (2011)
Baüerle, N., Rieder, U.: More risk-sensitive Markov decision processes. Math. Oper. Res. 39, 105–120 (2014)
Cao, X.R.: Semi-Markov decision problems and performance sensitivity analysis. IEEE Trans. Autom. Control 48, 758–769 (2003)
Cavazos-Cadena, R., Montes-De-Oca, R.: Optimal stationary policies in risk-sensitive dynamic programs with finite state space and nonnegative rewards. Appl. Math. (Warsaw) 27, 167–185 (2000)
Cavazos-Cadena, R., Montes-De-Oca, R.: Nearly optimal policies in risk-sensitive positive dynamic programming on discrete spaces. Math. Meth. Oper. Res. 52, 133–167 (2000)
Chung, K.J., Sobel, M.J.: Discounted MDP’s: distribution functions and exponential utility maximization. SIAM J. Control Optim. 25, 49–62 (1987)
Ghosh, M.K., Saha, S.: Risk-sensitive control of continuous time Markov chains. Stochastics 86, 655–675 (2014)
Ghosh, M.K., Saha, S.: Non-stationary semi-Markov decision processes on a finite horizon. Stoch. Anal. Appl. 31, 183–190 (2013)
Guo, X., Liu, Q.L., Zhang, Y.: Finite horizon risk-sensitive continuous-time Markov decision processes with unbounded transition and cost rates. 4OR 17, 427–442 (2019)
Guo, X.P., Hernández-Lerma, O.: Continuous-Time Markov Decision Processes: Theory and Applications. Springer, Berlin (2009)
Hernández-Lerma, O., Lasserre, J.B.: Discrete-Time Markov Control Processes: Basic Optimality Criteria. Springer, New York (1996)
Howard, R.A., Matheson, J.E.: Risk-sensitive Markov decision processes. Manage. Sci. 18, 356–369 (1972)
Huang, Y.H., Guo, X.P.: Discounted semi-Markov decision processes with nonnegative costs. Acta Math. Sin. (Chinese Ser.) 53, 503–514 (2010)
Huang, Y.H., Guo, X.P.: Finite horizon semi-Markov decision processes with application to maintenance systems. Eur. J. Oper. Res. 212, 131–140 (2011)
Huang, Y.H., Guo, X.P.: Mean-variance problems for finite horizon semi-Markov decision processes. Appl. Math. Optim. 72, 233–259 (2015)
Huang, Y.H., Guo, X.P., Song, X.Y.: Performance analysis for controlled semi-Markov process. J. Optim. Theory Appl. 150, 395–415 (2011)
Huang, Y.H., Lian, Z.T., Guo, X.P.: Risk-sensitive semi-Markov decision processes with general utilities and multiple criteria. Adv. Appl. Probab. 50, 783–804 (2018)
Huang, X.X., Zou, X.L., Guo, X.P.: A minimization problem of the risk probability in first passage semi-Markov decision processes with loss rates. Sci. China Math. 58, 1923–1938 (2015)
Huo, H.F., Zou, X.L., Guo, X.P.: The risk probability criterion for discounted continuous-time Markov decision processes. Discrete Event Dyn. Syst. 27, 675–699 (2017)
Janssen, J., Manca, R.: Semi-Markov Risk Models for Finance, Insurance, and Reliability. Springer, New York (2006)
Jaquette, S.C.: A utility criterion for Markov decision processes. Manage. Sci. 23, 43–49 (1976)
Jaśkiewicz, A.: A note on negative dynamic programming for risk-sensitive control. Oper. Res. Lett. 36, 531–534 (2008)
Jaśkiewicz, A.: On the equivalence of two expected average cost criteria for semi Markov control processes. Math. Oper. Res. 29, 326–338 (2013)
Limnios, N., Oprisan, G.: Semi-Markov Processes and Reliability. Birkhäuser, Boston (2001)
Luque-Vásquez, F., Minjárez-Sosa, J.A.: Semi-Markov control processes with unknown holding times distribution under a discounted criterion. Math. Meth. Oper. Res. 61, 455–468 (2005)
Mamer, J.W.: Successive approximations for finite horizon semi-Markov decision processes with application to asset liquidation. Oper. Res. 34, 638–644 (1986)
Nollau, V.: Solution of a discounted semi-Markovian decision problem by successive overrelaxation. Optimization 39, 85–97 (1997)
Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, New York (1994)
Schäl, M.: Control of ruin probabilities by discrete-time investments. Math. Meth. Oper. Res. 70, 141–158 (2005)
Wei, Q.D.: Continuous-time Markov decision processes with risk-sensitive finite-horizon cost criterion. Math. Meth. Oper. Res. 84, 1–27 (2016)
Wei, Q.D., Guo, X.P.: New average optimality conditions for semi-Markov decision processes in Borel spaces. J. Optim. Theory Appl. 153, 709–732 (2012)
Wei, Q.D., Guo, X.P.: Constrained semi-Markov decision processes with ratio and time expected average criteria in Polish spaces. Optimization 64, 1593–1623 (2015)
Yushkevich, A.A.: On semi-Markov controlled models with average reward criterion. Theory Probab. Appl. 26, 808–815 (1982)
Zhang, Y.: Continuous-time Markov decision processes with exponential utility. SIAM J. Control Optim. 55, 1–24 (2017)
Acknowledgement
This work was supported by National Natural Science Foundation of China (Grant No. 11961005, 11801590); Foundation of Guangxi Educational Committee (Grant No. KY2019YB0369); Ph.D. research startup foundation of Guangxi University of Science and Technology (Grant No. 18Z06); Guangxi Natural Science Foundation Program (Grant No. 2020GXNSFAA297196).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Huo, H., Wen, X. (2021). First Passage Exponential Optimality Problem for Semi-Markov Decision Processes. In: Piunovskiy, A., Zhang, Y. (eds) Modern Trends in Controlled Stochastic Processes:. Emergence, Complexity and Computation, vol 41. Springer, Cham. https://doi.org/10.1007/978-3-030-76928-4_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-76928-4_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-76927-7
Online ISBN: 978-3-030-76928-4
eBook Packages: EngineeringEngineering (R0)