Skip to main content
SpringerLink
Log in

Uniform ergodicity of continuous-time controlled Markov chains: A survey and new results

  • Published:
Annals of Operations Research Aims and scope Submit manuscript

Abstract

We make a review of several variants of ergodicity for continuous-time Markov chains on a countable state space. These include strong ergodicity, ergodicity in weighted-norm spaces, exponential and subexponential ergodicity. We also study uniform exponential ergodicity for continuous-time controlled Markov chains, as a tool to deal with average reward and related optimality criteria. A discussion on the corresponding ergodicity properties is made, and an application to a controlled population system is shown.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. Some authors refer to a function satisfying (i)–(iii) above as a transition function, while they call a transition function which satisfies, in addition, the condition (iv) a standard transition function.

References

  • Anderson, W. J. (1991). Continuous-time Markov chains. New York: Springer.

    Book  Google Scholar 

  • Chung, K. L. (1967). Markov chains with stationary transition probabilities (2nd ed.). Berlin: Springer.

    Google Scholar 

  • Dai, J. G., & Meyn, S. P. (1995). Stability and convergence of moments for multiclass queueing networks via fluid limit models. IEEE Transactions on Automatic Control, 40, 1889–1904.

    Article  Google Scholar 

  • Dekker, R., Hordijk, A., & Spieksma, F. M. (1994). On the relation between recurrence and ergodicity properties in denumerable Markov decision chains. Mathematics of Operations Research, 19, 539–559.

    Article  Google Scholar 

  • Dieudonné, J. (1960). Foundations of modern analysis. New York: Academic Press.

    Google Scholar 

  • Down, D., Meyn, S. P., & Tweedie, R. L. (1995). Exponential and uniform ergodicity of Markov processes. Annals of Probability, 23, 1671–1691.

    Article  Google Scholar 

  • Fort, G., & Moulines, E. (2002). Polynomial ergodicity of Markov transition kernels. Stochastic Processes and Their Applications, 103, 57–99.

    Article  Google Scholar 

  • Fort, G., & Roberts, G. O. (2005). Subgeometric ergodicity of strong Markov processes. The Annals of Applied Probability, 15, 1565–1589.

    Article  Google Scholar 

  • Ganidis, H., Roynette, B., & Simonot, F. (1999). Convergence rate of semi-groups to their invariant probability. Stochastic Processes and Their Applications, 79, 243–263.

    Article  Google Scholar 

  • Glynn, P. W., & Meyn, S. P. (1996). A Lyapunov bound for solutions of the Poisson equation. Annals of Probability, 24, 916–931.

    Article  Google Scholar 

  • Guo, X. P., & Hernández-Lerma, O. (2003). Drift and monotonicity conditions for continuous-time controlled Markov chains with an average criterion. IEEE Transactions on Automatic Control, 48, 236–244.

    Article  Google Scholar 

  • Guo, X. P., & Hernández-Lerma, O. (2009). Continuous-time Markov decision processes: theory and applications. New York: Springer.

    Book  Google Scholar 

  • Guo, X. P., & Piunovskiy, A. (2011). Discounted continuous-time Markov decision processes with constraints: unbounded transition and loss rates. Mathematics of Operations Research, 36, 105–132.

    Article  Google Scholar 

  • Guo, X. P., & Rieder, U. (2006). Average optimality for continuous-time Markov decision processes in polish spaces. The Annals of Applied Probability, 16, 730–756.

    Article  Google Scholar 

  • Guo, X. P., Hernández-Lerma, O., & Prieto-Rumeau, T. (2006). A survey of recent results on continuous-time Markov decision processes. Top, 14, 177–261.

    Article  Google Scholar 

  • Hernández-Lerma, O., & Lasserre, J. B. (1999). Further topics on discrete-time Markov control processes. New York: Springer.

    Book  Google Scholar 

  • Jarner, S. F., & Roberts, G. O. (2002). Polynomial convergence rates of Markov chains. The Annals of Applied Probability, 12, 224–247.

    Article  Google Scholar 

  • Katehakis, M. N., & Derman, C. (1989). On the maintenance of systems composed of highly reliable components. Management Science, 35, 551–560.

    Article  Google Scholar 

  • Kingman, J. F. C. (1963). Ergodic properties of continuous-time Markov processes and their discrete skeletons. Proceedings of the London Mathematical Society, 13, 593–604.

    Article  Google Scholar 

  • Lefèvre, C. (1981). Optimal control of a birth and death epidemic process. Operations Research, 29, 971–982.

    Article  Google Scholar 

  • Liu, Y., & Hou, Z. (2008). Exponential and strong ergodicity for Markov processes with an application to queues. Chinese Annals of Mathematics. Ser. B, 29, 199–206.

    Article  Google Scholar 

  • Liu, Y., Zhang, H., & Zhao, Y. (2008). Computable strongly ergodic rates of convergence for continuous-time Markov chains. ANZIAM Journal, 49, 463–478.

    Article  Google Scholar 

  • Liu, Y., Zhang, H., & Zhao, Y. (2010). Subgeometric ergodicity for continuous-time Markov chains. Journal of Mathematical Analysis and Applications, 368, 178–189.

    Article  Google Scholar 

  • Lund, R. B., Meyn, S. P., & Tweedie, R. L. (1996). Computable exponential convergence rates for stochastically ordered Markov processes. The Annals of Applied Probability, 6, 218–237.

    Article  Google Scholar 

  • Mao, Y. H. (2002). Strong ergodicity for Markov processes by coupling methods. Journal of Applied Probability, 39, 839–852.

    Article  Google Scholar 

  • Mao, Y. H. (2004). The eigentime identity for continuous-time ergodic Markov chains. Journal of Applied Probability, 41, 1071–1080.

    Article  Google Scholar 

  • Mao, Y. H. (2006). Convergence rates in strong ergodicity for Markov processes. Stochastic Processes and Their Applications, 116, 1964–1976.

    Article  Google Scholar 

  • Meyn, S. P., & Tweedie, R. L. (1993a). Markov chains and stochastic stability. London: Springer.

    Book  Google Scholar 

  • Meyn, S. P., & Tweedie, R. L. (1993b). Stability of Markovian processes II: continuous-time processes and sampled chains. Advances in Applied Probability, 25, 487–517.

    Article  Google Scholar 

  • Meyn, S. P., & Tweedie, R. L. (1993c). Stability of Markovian processes III: Foster-Lyapunov criteria for continuous-time processes. Advances in Applied Probability, 25, 518–548.

    Article  Google Scholar 

  • Prieto-Rumeau, T. (2006). Blackwell optimality in the class of Markov policies for continuous-time controlled Markov chains. Acta Applicandae Mathematicae, 92, 77–96.

    Article  Google Scholar 

  • Prieto-Rumeau, T., & Hernández-Lerma, O. (2005). The Laurent series, sensitive discount and Blackwell optimality for continuous-time controlled Markov chains. Mathematical Methods of Operations Research, 61, 123–145.

    Article  Google Scholar 

  • Prieto-Rumeau, T., & Hernández-Lerma, O. (2006). Bias optimality for continuous-time controlled Markov chains. SIAM Journal on Control and Optimization, 45, 51–73.

    Article  Google Scholar 

  • Prieto-Rumeau, T., & Hernández-Lerma, O. (2008). Ergodic control of continuous-time Markov chains with pathwise constraints. SIAM Journal on Control and Optimization, 47, 1888–1908.

    Article  Google Scholar 

  • Prieto-Rumeau, T., & Hernández-Lerma, O. (2012). Selected topics on continuous-time controlled Markov chains and Markov games. London: Imperial College Press.

    Google Scholar 

  • Prieto-Rumeau, T., & Lorenzo, J. M. (2010). Approximating ergodic average reward continuous-time controlled Markov chains. IEEE Transactions on Automatic Control, 55, 201–207.

    Article  Google Scholar 

  • Roberts, G. O., & Tweedie, R. L. (2000). Rates of convergence of stochastically monotone and continuous time Markov models. Journal of Applied Probability, 37, 359–373.

    Article  Google Scholar 

  • Tuominen, P., & Tweedie, R. L. (1994). Subgeometric rates of convergence of f-ergodic Markov chains. Advances in Applied Probability, 26, 775–798.

    Article  Google Scholar 

  • Ye, L., Guo, X. P., & Hernández-Lerma, O. (2008). Existence and regularity of a nonhomogeneous transition matrix under measurability conditions. Journal of Theoretical Probability, 21, 604–627.

    Article  Google Scholar 

  • Zhang, Y. H. (2001). Strong ergodicity for single birth processes. Journal of Applied Probability, 38, 270–277.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Statistics, UNED, Madrid, Spain

    Tomás Prieto-Rumeau

  2. Department of Mathematics, CINVESTAV-IPN, Mexico, DF, Mexico

    Onésimo Hernández-Lerma

Authors
  1. Tomás Prieto-Rumeau
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Onésimo Hernández-Lerma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tomás Prieto-Rumeau.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Prieto-Rumeau, T., Hernández-Lerma, O. Uniform ergodicity of continuous-time controlled Markov chains: A survey and new results. Ann Oper Res 241, 249–293 (2016). https://doi.org/10.1007/s10479-012-1184-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10479-012-1184-4

Keywords

Search

Navigation