Markov Chains

Abstract

This chapter deals with a particular class of stochastic models that form a cornerstone of this book. Stochastic models are widely used to describe phenomena that change randomly as time progresses.We focus onMarkov chains, as simple and adequate models for many such phenomena. More precise, we cover discrete—as well as continuous-time Markov chains. We discuss details of their analysis to set the ground for the requirements of later chapters, and introduce useful equivalence relations for both types of models. These relations are defined in the style of bisimilarity and are akin to the notion lumpability on Markov chains. Furthermore, we present efficient algorithms to compute these relations, which, as a side result, can be used to compute the ‘best possible’ lumping of a given Markov chain.