Skip to main content

Controlled Markov Chains


In Chap. 7 we considered Markov chains as a means to model stochastic DES for which explicit closed-form solutions can be obtained. Then, in Chap. 8, we saw how special classes of Markov chains (mostly, birth–death chains) can be used to model queueing systems. We pointed out, however, that queueing theory is largely “descriptive” in nature; that is, its main objective is to evaluate the behavior of queueing systems operating under a particular set of rules. On the other hand, we are often interested in “prescriptive” techniques, based on which we can make decisions regarding the “best” way to operate a system and ultimately control its performance. In this chapter, we describe some such techniques for Markov chains. Our main objective is to introduce the framework known as Markov Decision Theory, and to present some key results and techniques which can be used to control DES modeled as Markov chains. At the heart of these techniques is dynamic programming, which has played a critical role in both deterministic and stochastic control theory since the 1960s. The material in this chapter is more advanced than that of previous ones, it involves some results that were published in the research literature fairly recently, and it demands slightly higher mathematical sophistication. The results, however, should be quite gratifying for the reader, as they lead to the solution of some basic problems from everyday life experience, or related to the basic operation of some very familiar DES whose predominant characteristic is resource contention. The last part of this chapter is devoted to some such problems pertaining to queueing systems.

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-72274-6_9
  • Chapter length: 57 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
USD   119.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-72274-6
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Hardcover Book
USD   159.99
Price excludes VAT (USA)
Figure 9.1:
Figure 9.2:
Figure 9.3:
Figure 9.4:
Figure 9.5:
Figure 9.6:
Figure 9.7:
Figure 9.8:
Figure 9.9:
Figure 9.10:
Figure 9.11:
Figure 9.12:
Figure 9.13:

Author information

Authors and Affiliations


Corresponding author

Correspondence to Christos G. Cassandras .

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this chapter

Verify currency and authenticity via CrossMark

Cite this chapter

Cassandras, C.G., Lafortune, S. (2021). Controlled Markov Chains. In: Introduction to Discrete Event Systems. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-72272-2

  • Online ISBN: 978-3-030-72274-6

  • eBook Packages: Computer ScienceComputer Science (R0)