Abstract
This paper provides a characterization of the optimal average cost function, when the long-run (risk-sensitive) average cost criterion is used. The Markov control model has a denumerable state space with finite set of actions, and the characterization presented is given in terms of a system of local Poisson equations, which gives as a by-product the existence of an optimal stationary policy.
Similar content being viewed by others
References
Cavazos-Cadena, R., Hernández-Hernández, D.: A system of Poisson equations for a non-constant Varadhan functional on finite state space. Appl. Math. Optim. 53, 101–119 (2006)
Cavazos-Cadena, R., Hernández-Hernández, D.: Local Poisson equations associated with the Varadhan functional. Asymptot. Anal. 96, 023–050 (2015)
Alanís-Durán, A., Cavazos-Cadena, R.: An optimality system for finite average Markov decision chains under risk aversion. Kybernetika 48, 83–104 (2012)
Howard, A.R., Matheson, J.E.: Risk sensitive Markov decision processes. Manag. Sci. 18, 356–369 (1972)
Gantmakher, F.R.: The Theory of Matrices. Chelsea, London (1959)
Meyer, C.D.: Matrix Analysis and Applied Linear Algebra. SIAM, Philadelphia (2000)
Fleming, W.H., Hernández-Hernández, D.: Risk sensitive control of finite state machines on a infinite horizon I. SIAM J. Control Optim. 35, 1790–1810 (1997)
Bielecki, T., Hernández-Hernández, D., Pliska, R.: Risk sensitive control of finite state Markov chains in discrete time, with applications to portfolio management. Math. Methods Oper. Res. 50, 167–188 (1999)
Hernández-Hernández, D., Marcus, S.I.: Risk sensitive control of Markov processes in countable state space. Syst. Control Lett. 29, 147–155 (1996). Corrigendum. 34, 105–106 (1999)
Cavazos-Cadena, R., Fernández-Gaucherand, E.: Controlled Markov chains with risk-sensitive criteria: average cost, optimality equations, and optimal solutions. Math. Methods Oper. Res. 49, 299–324 (1999)
Cavazos-Cadena, R., Fernández-Gaucherand, E.: Risk sensitive control in communicating average Markov decision chains. In: Dror, P., L’Ecuyer, P., Szidarovsky, F. (eds.) Modelling Uncertainty: An Examination of Stochastic Theory, Methods and Applications, pp. 525–544. Kluwer, Boston (2002)
Di Masi, G.B., Stettner, L.: Risk sensitive control of discrete time Markov processes with infinite horizon. SIAM J. Control Optim. 38, 61–78 (1999)
Cavazos-Cadena, R.: Solution to the risk sensitive average cost optimality equation in a class of Markov decision processes with finite state space. Math. Methods Optim. Res. 57, 263–285 (2003)
Sladký, K.: Growth rates and average optimality in risk sensitive Markov decision chains. Kybernetika 44, 205–226 (2008)
Cavazos-Cadena, R., Salem-Silva, F.: The discounted method and equivalence of average criteria for risk sensitive Markov decision processes on Borel space. Appl. Math. Optim. 61, 167–190 (2009)
Di Masi, G.B., Stettner, L.: Infinite horizon risk sensitive control of discrete time Markov processes with small risk. Syst. Control Lett. 40, 15–20 (2000)
Di Masi, G.B., Stettner, L.: Infinite horizon risk sensitive control of discrete time Markov processes under minorization property. SIAM J. Control Optim. 46, 231–252 (2007)
Sladký, K.: Bounds on discrete dynamic programming recursions I. Kybernetika 16, 526–547 (1980)
Sladký, K., Montes-de-Oca, R.: Risk-sensitive average optimality in Markov decision chains. In: Kalcsiscs, J., Nickel, S. (eds.) Operations Research Proceedings, vol. 2007, pp. 69–74. Springer, Berlin (2008)
Zijim, W.H.M.: Nonnegative Matrices in Dynamic Programming. Mathematical Centre Tract, Amsterdam (1983)
Rothblum, U.G., Whittle, P.: Growth optimality for branching Markov desicion chains. Math. Oper. Res. 7, 582–601 (1982)
Hernández-Lerma, O., Lasserre, J.B.: Discrete-Time Markov Control Processes: Basic Optimality Criteria. Springer, New York (1996)
Ash, R.B.: Probability and Measure Theory. Academic Press, New York (2000)
Billingsley, P.: Probability and Measure. Wiley, New York (1995)
Acknowledgements
DHH was partially supported by CONACYT Grant 254166.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by Alan Bensoussan.
Rights and permissions
About this article
Cite this article
Hernández Hernández, D., Hernández Bustos, D. Local Poisson Equations Associated with Discrete-Time Markov Control Processes. J Optim Theory Appl 173, 1–29 (2017). https://doi.org/10.1007/s10957-017-1076-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10957-017-1076-5