Continuous Time Markov Chains

Abstract

WE TURN now to the continuous time version of the Markov property. Some of the simplicity of Chapter 2 is retained, because we assume the state space S is discrete. Usually we can suppose that S = {0, 1, … }. The succession of states visited still follows a discrete parameter Markov chain but now the flow of time is perturbed by exponentially distributed holding times in each state. An easy generalization of the dissection argument of Chapter 2 shows that the process regenerates at return times to a fixed reference state, so renewal theory and regenerative processes are useful.