Skip to main content
SpringerLink
Log in

Existence and Regularity of a Nonhomogeneous Transition Matrix under Measurability Conditions

  • Published:
Journal of Theoretical Probability Aims and scope Submit manuscript

Abstract

This paper is about the existence and regularity of the transition probability matrix of a nonhomogeneous continuous-time Markov process with a countable state space. A standard approach to prove the existence of such a transition matrix is to begin with a continuous (in t≥0) and conservative matrix Q(t)=[q ij (t)] of nonhomogeneous transition rates q ij (t) and use it to construct the transition probability matrix. Here we obtain the same result except that the q ij (t) are only required to satisfy a mild measurability condition, and Q(t) may not be conservative. Moreover, the resulting transition matrix is shown to be the minimum transition matrix, and, in addition, a necessary and sufficient condition for it to be regular is obtained. These results are crucial in some applications of nonhomogeneous continuous-time Markov processes, such as stochastic optimal control problems and stochastic games, and this was the main motivation for this work.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Ash, R.B.: Probability and Measure Theory, 2nd edn. Academic, London (2000)

    MATH  Google Scholar 

  2. Breuer, B.: From Markov Jump Processes to Spatial Queues. Kluwer, Dordrecht (2003)

    MATH  Google Scholar 

  3. Feller, W.: On the integro-differential equations of purely discontinuous Markov processes. Trans. Am. Math. Soc. 48, 488–515 (1940)

    Article  MATH  MathSciNet  Google Scholar 

  4. Gikhman, I.I., Skorokhod, A.V.: Introduction to the Theory of Random Processes. Dover, Mineola (1996) (this is a reprint of the 1969 edition published by Saunders, Philadelphia, PA)

    Google Scholar 

  5. Guo, X.P., Hernández-Lerma, O.: Nonzero-sum games for continuous-time Markov chains with unbounded discounted payoffs. J. Appl. Probab. 42, 303–320 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  6. Guo, X.P., Hernández-Lerma, O.: Zero-sum games for continuous-time Markov chains with unbounded transition and average payoff rates. J. Appl. Probab. 40, 327–345 (2003)

    Article  MATH  MathSciNet  Google Scholar 

  7. Guo, X.P., Hernández-Lerma, O.: Drift and monotonicity conditions for continuous-time controlled Markov chains with an average criterion. IEEE Trans. Autom. Control 48, 236–245 (2003)

    Article  Google Scholar 

  8. Guo, X.P., Hernández-Lerma, O.: Continuous–time controlled Markov chains. Ann. Appl. Prob. 13, 363–388 (2003)

    Article  MATH  Google Scholar 

  9. Guo, X.P., Zhu, W.P.: Denumerable-state continuous-time Markov decision processes with unbounded transition and reward rates under the discounted criterion. J. Appl. Probab. 39, 233–250 (2002)

    Article  MATH  MathSciNet  Google Scholar 

  10. Hu, D.H.: Markov Process Theory with Countable State. Wuhan University Press, Wuhan (1983) (in Chinese)

    Google Scholar 

  11. Hu, Q.Y.: Nonstationary continuous time Markov decision processes with the expected total reward criterion. Optimization 36, 181–189 (1996)

    Article  MATH  MathSciNet  Google Scholar 

  12. Kakumanu, P.: Continuous time Markovian decision process with average return criterion. J. Math. Anal. Appl. 52, 173–188 (1975)

    Article  MATH  MathSciNet  Google Scholar 

  13. Liu, J.Y., Hu, Q.Y., Wang, J.M.: The foundational assumption of continuous-time Markov decision process. Acta Math. Appl. Sin. 27, 756–759 (2004) (in Chinese)

    MathSciNet  Google Scholar 

  14. Puterman, M.L.: Markov Decision Processes. Wiley, New York (1994)

    MATH  Google Scholar 

  15. Sennott, L.I.: Stochastic Dynamic Programming and the Control of Queuing Systems. Wiley, New York (1999)

    Google Scholar 

  16. Sinha, S., Jana, S.: Semi-infinite semi-Markov stochastic games. Opsearch 41, 278–290 (2004)

    MATH  MathSciNet  Google Scholar 

  17. Stidham, S., Weber, R.: A survey of Markov decision models for the control of networks of queues. Queuing Syst. 13, 291–314 (1993)

    Article  MATH  MathSciNet  Google Scholar 

  18. Wu, C.B.: Continuous time Markov decision process with unbounded reward and nonuniformly bounded transition rates under discounted criterion. Acta Math. Appl. Sin. 20, 196–208 (1997) (in Chinese)

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. The School of Mathematics and Computational Science, Zhongshan University, Guangzhou, China

    Liuer Ye & Xianping Guo

  2. Departamento de Matemáticas, CINVESTAV-IPN, A.Postal 14-740, D.F. 07000, México, Mexico

    Onésimo Hernández-Lerma

Authors
  1. Liuer Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xianping Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Onésimo Hernández-Lerma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Onésimo Hernández-Lerma.

Additional information

Supported by NSFC and RFDP.

The research of O. Hernández-Lerma was partially supported by CONACYT grant 45693-F.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ye, L., Guo, X. & Hernández-Lerma, O. Existence and Regularity of a Nonhomogeneous Transition Matrix under Measurability Conditions. J Theor Probab 21, 604–627 (2008). https://doi.org/10.1007/s10959-008-0163-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10959-008-0163-9

Keywords

Mathematics Subject Classification (2000)

Search

Navigation