Skip to main content

Markov Chains on Continuous State Space

  • Chapter
  • 2089 Accesses

Abstract

In this chapter, we discuss Markov chains on continuous state space. We first analyze a discrete-time Markov chain on continuous state space, and then discuss a discrete-time Markov chain on a bivariate state space. Applying the censoring technique, we provide expression for the RG-factorizations, which are used to derive the stationary probability of the Markov chain. Further, we consider a continuous-time Markov chain on continuous state space. Specifically, we deal with a continuous-time level-dependent QBD process with continuous phase variable, and provide orthonormal representations for the R-, U- and G-measures, which lead to the matrix-structured computation of the stationary probability. As an application, we introduce continuousphase type (CPH) distribution and continuous-phase Markovian arrival process (CMAP), and then analyze a CMAP/CPH/1 queue. Finally, we study a piecewise deterministic Markov process, which is applied to deal with more general queues such as the GI/G/c queue.

Keywords

  • Markov chains on continuous state space
  • QBD process with continuous phase variable
  • orthonormal representation
  • continuous-phase type distribution
  • continuous-phase Markovian arrival process
  • piecewise deterministic Markov process

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-642-11492-2_5
  • Chapter length: 72 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   149.00
Price excludes VAT (USA)
  • ISBN: 978-3-642-11492-2
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Hardcover Book
USD   189.00
Price excludes VAT (USA)

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Anderson W.J. (1991). Continuous-time Markov chains: An applications-oriented approach, Springer-Verlag

    Google Scholar 

  2. Artalejo J.R. and A. Gómez-Corral (2008). Retrial Queueing Systems: A Computational Approach, Springer

    Google Scholar 

  3. Athreya K.B. and P. Ney (1978). A new approach to the limit theory of recurrent Markov chains. Transactions of the American Mathematical Society 245: 493–501

    MATH  CrossRef  MathSciNet  Google Scholar 

  4. Athreya K.B. and P. Ney (1982). A renewal approach to the Perron-Frobenius theory of non-negative kernels on general state space. Mathematische Zeitschrift 179: 507–529

    MATH  CrossRef  MathSciNet  Google Scholar 

  5. Breuer L. (2002). On Markov-additive jump process. Queueing Systems 40: 75–91

    MATH  CrossRef  MathSciNet  Google Scholar 

  6. Breuer L. (2003). Transient and stationary distributions for the GI/G/k queue with Lebesguedominated inter-arrival time distribution. Queueing Systems 45: 47–57

    MATH  CrossRef  MathSciNet  Google Scholar 

  7. Breuer L. (2008). Continuity of the M/G/c queue. Queueing Systems 58: 321–331

    MATH  CrossRef  MathSciNet  Google Scholar 

  8. Costa O.L.V. (1990). Stationary distributions for piecewise-deterministic Markov processes. Journal of Applied Probability 27: 60–73

    MATH  CrossRef  MathSciNet  Google Scholar 

  9. Davis M.H.A. (1984). Piecewise-deterministic Markov processes: A general class of nondiffusion stochastic models. Journal of the Royal Statistical Society (Series B) 46: 353–388

    MATH  Google Scholar 

  10. Dempster M.A.H. (1991). Optimal control of piecewise-deterministic Markov processes. In Applied Stochastic Analysis, M.H.A. Davis and R.J. Elliott (eds), Gordon and Breach: New York, 303–325

    Google Scholar 

  11. Dufour F. and O.L.V. Costa (1999). Stability of piecewise deterministic Markov processes. SIAM Journal on Control and Optimization 37: 1483–1502

    MATH  CrossRef  MathSciNet  Google Scholar 

  12. Finch P.D. (1963). A limit theorem for Markov chains with continuous state space. Journal of the Australian Mathematical Society 3: 351–358

    MATH  CrossRef  MathSciNet  Google Scholar 

  13. Gail H.R., S.L. Hantler and B.A. Taylor (1994). Solutions of the basic matrix equation for M/G/1 type and G/M/1 type Markov chains. Stochastic Models 10: 1–43

    MATH  CrossRef  MathSciNet  Google Scholar 

  14. Gugerli U.S. (1986). Optimal stopping of piecewise deterministic Markov processes. Stochastics 19: 221–236

    MATH  MathSciNet  Google Scholar 

  15. Hernández-Lerma O. and J.B. Lasserre (2003). Markov chains and invariant probabilities. Progress in Mathematics, 211. Birkhäuser Verlag, Basel, 2003

    MATH  Google Scholar 

  16. Hou Z.T. and G.X. Liu (2005). Markov Skeleton Processes and their Applications, Science Press and International Press

    Google Scholar 

  17. Kemeny J.G., J.L. Snell and A.W. Knapp (1976). Denumerable Markov Chains (Second Edition), Springer-Verlag: New York

    MATH  Google Scholar 

  18. Li Q.L. and C. Lin (2006). Continuous-time QBD processes with continuous phase variable. Computers & Mathematics with Applications 52: 1483–1510

    MATH  CrossRef  MathSciNet  Google Scholar 

  19. Li Q.L., G.H. Wang and Y. Zhou (2004). Symmetric PH and EP distributions and their applications to the probability Hough transform. Computers & Mathematics with Applications 47: 823–844

    MATH  CrossRef  MathSciNet  Google Scholar 

  20. Liu G.X. (1998). Piecewise deterministic Markov processes, Ph.D. Thesis, Changsha Railway University, Changsha, P.R. China

    Google Scholar 

  21. Liu G.X. (2002). Piecewise deterministic Markov processes and semi-dynamic systems. In Markov Processes and Control Markov Chains, J.A. Filar and Z.T. Hou (eds), Kluwer, 93–107

    Google Scholar 

  22. Meyn S.P. and R.L. Tweedie (1993). Markov chains and stochastic stability. Springer-Verlag, 1993

    Google Scholar 

  23. Neuts M.F. (1981). Matrix-Geometric Solutions in Stochastic Models-An Algorithmic Approach, The Johns Hopkins University Press: Baltimore

    MATH  Google Scholar 

  24. Nielsen B.F. and V. Ramaswami (1997). A computational framework for a quasi birth and death process with a continuous phase variable. In ITC-15, V. Ramaswami and P.E. Wirth (eds.), Elsevier Science B.V., 477–486

    Google Scholar 

  25. Prieto-Rumeau T. and O. Hernández-Lerma (2008). Ergodic control of continuous-time Markov chains with pathwise constraints. SIAM Journal on Control and Optimization 47: 1888–1908

    MATH  CrossRef  MathSciNet  Google Scholar 

  26. Sengupta B. (1986). An operator geometric solution for a buffer occupancy problem. Stochastic Models 2: 251–271

    MATH  CrossRef  Google Scholar 

  27. Tweedie R.L., Operator-geometric stationary distributions for Markov chains, with application to queueing models. Advances in Applied Probability 14: 368–391

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Rights and permissions

Reprints and Permissions

Copyright information

© 2010 Tsinghua University Press, Beijing and Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Li, QL. (2010). Markov Chains on Continuous State Space. In: Constructive Computation in Stochastic Models with Applications. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-11492-2_5

Download citation