Abstract
In the previous chapter, we presented the Generalized Semi-Markov Process (GSMP) framework as a means of modeling stochastic DES. By allowing event clocks to tick at varying speeds, we also provided an extension to the basic GSMP. In addition, we introduced the Poisson process as a basic building block for a class of stochastic DES which possess the Markov (memoryless) property. Thus, we obtained the class of stochastic processes known as Markov chains, which we will study in some detail in this chapter. It should be pointed out that the analysis of Markov chains provides a rich framework for studying many DES of practical interest, ranging from gambling and the stock market to the design of “high-tech” computer systems and communication networks.
The main characteristic of Markov chains is that their stochastic behavior is described by transition probabilities of the form P[X(tk+1) = x′ ǀ X(t k ) = x] for all state values x, x′ and t k ≤ t k +1. Given these transition probabilities and a distribution for the initial state, it is possible to determine the probability of being at any state at any time instant. Describing precisely how to accomplish this and appreciating the difficulties involved in the process are the main objectives of this chapter.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Selected References
∎ Asmussen, S., Applied Probability and Queues, Wiley, New York, 1987.
∎ Bertsekas, D.P., Dynamic Programming: Deterministic and Stochastic Models, Prentice-Hall, Englewood Cliffs, NJ, 1987.
∎ Bertsekas, D.P., Dynamic Programming and Optimal Control, Vols. 1 and 2, Athena Scientific, Belmont, MA, 1995.
∎ Cinlar, E., Introduction to Stochastic Processes, Prentice-Hall, Englewood Cliffs, NJ, 1975.
∎ Clarke, A.B., and R.L. Disney, Probability and Random Processes, Wiley, New York, 1985.
∎ Gallager, R.G., Discrete Stochastic Processes, Kluwer Academic Publishers, Boston, 1996.
∎ Hoel, P.G., S.C. Port, and C.J. Stone, Introduction to Stochastic Processes, Houghton Mifflin, Boston, 1972.
∎ Kleinrock, L., Queueing Systems, Volume I: Theory, Wiley, New York, 1975.
∎ Parzen, E., Stochastic Processes, Holden-Day, San Francisco, 1962.
∎ Trivedi, K.S., Probability and Statistics with Reliability, Queuing and Computer Science Applications, Second Edition, Wiley, New York, 2001.
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2008 Springer Science+Business Media, LLC
About this chapter
Cite this chapter
(2008). Markov Chains. In: Cassandras, C.G., Lafortune, S. (eds) Introduction to Discrete Event Systems. Springer, Boston, MA. https://doi.org/10.1007/978-0-387-68612-7_7
Download citation
DOI: https://doi.org/10.1007/978-0-387-68612-7_7
Publisher Name: Springer, Boston, MA
Print ISBN: 978-0-387-33332-8
Online ISBN: 978-0-387-68612-7
eBook Packages: EngineeringEngineering (R0)