Markov Processes with Countable State Spaces

Abstract

AMarkov processwith a countable state space is a special case of a semi-Markov process in which, first, the interval between successive transitions has an exponential distribution, and second, that interval is independent of the next state. Thus, we can take the set of possible states as {0, 1, 2,…} and the process as {X(t),t≥0} where for each real t≥0,X(t) is the state of the process at time t. The random variables S₁, S₂,… denote the successive epochs at which the process makes state transitions, and X_ndenotes the state entered at time S_n, i.e., X_n = X(S_n) and X(t) = X_n for S_n≤t≤S_n+1. Let S₀=0, and let X₀= X(0) = X(S₀) denote the initial state. Theembedded Markov chainhas transition probabilities {P_ij, i≥0, j≥0}, and we assume that P_ii=0 for all i (i.e., there are no self transitions). The assumption P_ii=0 will be removed later when we talk about uniformized Markov processes. The intervals U_n=S_n−S_n− between successive transition epochs satisfy

$$P({{U}_{n}} \leqslant x|{{X}_{{n - 1}}} = i,{{X}_{n}} = j\} = 1 - \exp ( - {{\nu }_{i}}x)$$

(1)

where, for each i, v_i is a positive number called thetransition rateout of state i. Conditional on X_n-1, the interval U_nis independent of X_nand also independent of all earlier inter-transition intervals and states.