Markov Chains and Applications
In many applications, successive observations of a process, say X 1, X 2,…, have an inherent time component associated with them. For example, the X i could be the state of the weather at a particular location on the i th day, counting from some fixed day. In a simplistic model, the state of the weather could be “dry” or “wet,” quantified as, say, 0 and 1. It is hard to believe that in such an example, the sequence X 1, X 2,… could be mutually independent. The question then arises how to model the dependence among the X i ′s. Probabilists have numerous dependency models. A particular model that has earned a very special status is called the Markov chain. In a Markov chain model, we assume that the future, given the entire past and the present state of a process, depends only on the present state. In the weather example, suppose we want to assign a probability that tomorrow, say March 10, will be dry, and suppose that we have available to us the precipitation history for each day from January 1 to March 9. In a Markov chain model, our probability that March 10 will be dry will depend only on the state of the weather on March 9, even though the entire past precipitation history was available to us. As simple as it sounds, Markov chains are enormously useful in applications, perhaps more than any other specific dependency model. They also are independently relevant to statistical computing in very important ways. The topic has an incredibly rich and well-developed theory, with links to many other topics in probability theory. Familiarity with basic Markov chain terminology and theory is often considered essential for anyone interested in studying statistics and probability.We present an introduction to basic Markov chain theory in this chapter.
KeywordsMarkov Chain Stationary Distribution Markov Chain Model Transition Probability Matrix Stationary Markov Chain
Unable to display preview. Download preview PDF.
- Bhattacharya, R.N. and Waymire, E. (2009). Stochastic Processes with Applications, SIAM, Philadelphia.Google Scholar
- Brémaud, P. (1999). Markov Chains, Gibbs Fields, Monte Carlo, and Queues, Springer, New York.Google Scholar
- Diaconis, P. (1988). Group Representations in Probability and Statistics, IMS Lecture Notes and Monographs Series, Hayward, CA.Google Scholar
- Feller, W. (1968). An Introduction to Probability Theory, with Applications, Wiley, New York.Google Scholar
- Freedman, D. (1975). Markov Chains, Holden-Day, San Francisco.Google Scholar
- Isaacson, D. and Madsen, R. (1976). Markov Chains, Theory and Applications, Wiley, New York.Google Scholar
- Kemperman, J. (1950). The General One-Dimensional Random Walk with Absorbing Barriers, Geboren Te, Amsterdam.Google Scholar
- Meyn, S. and Tweedie, R. (1993). Markov Chains and Stochastic Stability, Springer, New York.Google Scholar
- Norris, J. (1997). Markov Chains, Cambridge University Press, Cambridge.Google Scholar
- Seneta, E. (1981). Nonnegative Matrices and Markov Chains, Springer-Verlag, New York.Google Scholar
- Stirzaker, D. (1994). Elementary Probability, Cambridge University Press, Cambridge.Google Scholar