Recurrence and Transience of Markov Chains
The general problem of convergence in distribution of Markov chains with a countable state space is investigated in this chapter. For the GI/GI/1 queue in Chapter 2, the convergence in distribution of the Markov chain (W n ) has been obtained by using an explicit representation of the random variable W n as a functional of the random walk associated with the interarrival times and the service times. Markov chains describing the behavior of most queueing systems cannot, in general, be represented in such a simple way. In this chapter, simple criteria are given to determine whether a given Markov chain is ergodic or transient. The main results are Theorem 8.6 for ergodicity and Theorem 8.10 for transience. In practice, these results can be used in many applications. These criteria can be seen as an extension, in a probabilistic setting, of a classical stability result of ordinary differential equations due to Lyapunov[Lia07] in 1892. In a stochastic context, the first results of this type are apparently due to Khasminskii for diffusions (see Khasminskii ). The stability criterion by Lyapunov is recalled at the end of the chapter (see Hirsch and Smale  for a detailed presentation of these questions). In Chapter 9, an important scaling method is introduced that will give a more precise picture of the relation between the stability of ordinary differential equations and the stability properties of Markov
KeywordsMarkov Chain Markov Process Lyapunov Function Markov Property Strong Markov Property
Unable to display preview. Download preview PDF.
- [Lia07]A.M. Liapunov, Problème général de la stabilité du mouvement, Annales de la Faculté des Sciences de lUniversité de Toulouse 9 (1907), 203–475.Google Scholar
- [Fil89]Y. Filonov, A criterion for the ergodicity of discrete homogeneous Markov chains, Akademiya nauk Ukrainskoi SSR. Institut Matematiki. Ukrainskii Matematicheskii Zhurual 41 (1989), no. 10, 1421–1422.Google Scholar
- [Lam60]J. Lamperti, Criteria foe the recurrence or transience of stochastic process I., Journal of Mathematical Analysis and Applications 1 (1960), 314–330.Google Scholar