Discrete-Time Markov Chains
Applied probability thrives on models. Markov chains are one of the richest sources of good models for capturing dynamical behavior with a large stochastic component [23, 24, 59, 80, 106, 107, 118]. In this chapter we give a few examples and a quick theoretical overview of discrete-time Markov chains. The highlight of our theoretical development, Proposition 7.4.1, relies on a coupling argument. Because coupling is one of the most powerful and intuitively appealing tools available to probabilists, we examine a few of its general applications as well. We also stress reversible Markov chains. Reversibility permits explicit construction of the long-run or equilibrium distribution of a chain when such a distribution exists. Chapter 8 will cover continuous-time Markov chains.
KeywordsMarkov Chain Simulated Annealing Equilibrium Distribution Detailed Balance Transition Probability Matrix
Unable to display preview. Download preview PDF.