Markov Chains with Countably Infinite State Spaces

Abstract

Markov chains with a countably infinite state space exhibit some types of behavior not possible for chains with a finite state space. Figure 5.1 helps explain how these new types of behavior arise. If p > 1/2, then transitions to the right occur with higher frequency than transitions to the left. Thus, reasoning heuristically, we expect X_n to be large for large n. This means that, given X₀ = 0, the probability P ⁿ_0j should go to zero for any fixed j with increasing n. If one tried to define the steady state probability of state j as lim_n→∞P ⁿ_0j ,then this limit would be 0 for all j. These probabilities would not sum to 1, and thus would not correspond to a limiting distribution. Thus we say that a steady state does not exist. In more heuristic terms, the state keeps increasing forever. The truncation of figure 5.1 to k states is analyzed in exercise 4.3. The solution there defines ρ=p/q and shows that π₁=(1−ρ)ρ¹/(1−ρ^k) for ρ≠1 and π₁ = 1/k for ρ=1. For ρ<1 the limiting behavior as k → ∞ is π₁= (1−ρ)ρⁱ for ρ<1 and π_i=0 otherwise. In section 5.3 we analyze birth death Markov chains, of which figure 5.1 is an example, without first truncating the chain.