Abstract
In spite of their simplicity, Markov processes with countable state space (and discrete time) are interesting mathematical objects with which a variety of real-world phenomena can be modeled. We give an introduction to the basic concepts (Markov property, transition matrix, recurrence, transience, invariant distribution) and then study certain examples in more detail. For example, we show how to compute numerically very precisely, the expected number of returns to the origin of simple random walk on multidimensional integer lattices.
The connection with discrete potential theory will be investigated later, in Chap. 19. Some readers might prefer to skip the somewhat technical construction of general Markov processes in Sect. 17.1.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
L. Breiman, Probability (Addison-Wesley Publishing Company, Reading, MA, 1968)
P. Brémaud, Markov Chains. Texts in Applied Mathematics, vol. 31. Gibbs fields, Monte Carlo simulation, and queues (Springer, New York, 1999)
K.L. Chung, Markov Chains with Stationary Transition Probabilities. Die Grundlehren der mathematischen Wissenschaften, Bd. 104 (Springer, Berlin, 1960)
K.L. Chung, W.H.J. Fuchs, On the distribution of values of sums of random variables. Mem. Am. Math. Soc. 6, 1–12 (1951)
R.M. Dudley, Real Analysis and Probability. Cambridge Studies in Advanced Mathematics, vol. 74 (Cambridge University Press, Cambridge, 2002). Revised reprint of the 1989 original
R. Durrett, Probability: Theory and Examples, 4th edn. Cambridge Series in Statistical and Probabilistic Mathematics. (Cambridge University Press, Cambridge 2010)
W. Feller, An Introduction to Probability Theory and its Applications, vol. I, 3rd edn. (Wiley, New York, 1968)
H.-O. Georgii, Stochastics: Introduction to Probability Theory and Statistics. de Gruyter Lehrbuch, 2nd edn. (Walter de Gruyter & Co., Berlin, 2012)
A.L. Gibbs, F.E. Su, On choosing and bounding probability metrics. Int. Stat. Rev. 70(3), 419–435 (2002)
M.L. Glasser, I.J. Zucker, Extended Watson integrals for the cubic lattices. Proc. Nat. Acad. Sci. U.S.A. 74(5), 1800–1801 (1977)
G.R. Grimmett, D.R. Stirzaker, Probability and Random Processes, 3rd edn. (Oxford University Press, New York, 2001)
O. Häggström, Finite Markov Chains and Algorithmic Applications. London Mathematical Society Student Texts, vol. 52 (Cambridge University Press, Cambridge, 2002)
G.S. Joyce, Singular behaviour of the lattice Green function for the d-dimensional hypercubic lattice. J. Phys. A 36(4), 911–921 (2003)
O. Kallenberg, Foundations of Modern Probability, 2nd edn. Probability and Its Applications (Springer, New York/Berlin, 2002)
L.V. Kantorovič, G.Š. Rubinšteı̆n, On a space of completely additive functions. Vestnik Leningrad Univ. 13(7), 52–59 (1958)
J.G. Kemeny, J.L. Snell, Finite Markov Chains. Undergraduate Texts in Mathematics (Springer, New York, 1976). Reprinting of the 1960 original
A. Klenke, L. Mattner, Stochastic ordering of classical discrete distributions. Adv. in Appl. Probab. 42(2), 392–410 (2010)
S.P. Meyn, R.L. Tweedie, Markov Chains and Stochastic Stability. Communications and Control Engineering Series (Springer, London, 1993)
A. Müller, D. Stoyan, Comparison Methods for Stochastic Models and Risks. Wiley Series in Probability and Statistics (Wiley, Chichester, 2002)
J.R. Norris, Markov Chains. Cambridge Series in Statistical and Probabilistic Mathematics (Cambridge University Press, Cambridge, 1998). Reprint of the 1997 edition
E. Nummelin, General Irreducible Markov Chains and Nonnegative Operators. Cambridge Tracts in Mathematics, vol. 83 (Cambridge University Press, Cambridge, 1984)
G. Pólya, Über eine Aufgabe der Wahrscheinlichkeitsrechnung betreffend die Irrfahrt im Straßennetz. Math. Ann. 84, 149–160 (1921)
D. Revuz, Markov Chains. North-Holland Mathematical Library, vol. 11, 2nd edn. (North-Holland Publishing Co., Amsterdam, 1984)
E. Seneta, Non-negative Matrices and Markov Chains. Springer Series in Statistics (Springer, New York, 2006). Revised reprint of the second (1981) edition
F. Spitzer, Principles of Random Walks. Graduate Texts in Mathematics, vol. 34, 2nd edn. (Springer, New York, 1976)
V. Strassen, The existence of probability measures with given marginals. Ann. Math. Statist. 36, 423–439 (1965)
G.N. Watson, Three triple integrals. Q. J. Math. Oxford Ser. 10, 266–276 (1939)
S. Wright, Evolution in Mendelian populations. Genetics 16, 97–159 (1931)
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2020 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Klenke, A. (2020). Markov Chains. In: Probability Theory. Universitext. Springer, Cham. https://doi.org/10.1007/978-3-030-56402-5_17
Download citation
DOI: https://doi.org/10.1007/978-3-030-56402-5_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-56401-8
Online ISBN: 978-3-030-56402-5
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)