Encyclopedia of Operations Research and Management Science

2001 Edition
| Editors: Saul I. Gass, Carl M. Harris

Markov decision processes

Reference work entry
DOI: https://doi.org/10.1007/1-4020-0611-X_580

The finite-state, finite-action Markov decision process is a particularly simple and relatively tractable model of sequential decision making under uncertainty. It has been applied in such diverse fields as health care, highway maintenance, inventory, ma-chine maintenance, cash-flow management, and regulation of water reservoir capacity (Derman, 1970; Hernandez-Lermer, 1989; Ross, 1970; White, 1969). Here we present a definition of a Markov decision process and illustrate it with an example, followed by a discussion of the various solution procedures for several different types of Markov decision processes, all of which are based on dynamic programming (Bertsekas, 1987; Howard, 1971; Puterman, 1994; Sennott, 1999).


Let k ∈ {0, 1,..., K − 1} represent the kth stage or decision epoch, that is, when the kth decision must be selected; K < ∞ represents the planning horizon of the Markov decision process. Let sk be the state of the system to be con-trolled at stage k....
This is a preview of subscription content, log in to check access.


  1. [1]
    Bertsekas, D.P. (1987). Dynamic Programming: Deterministic and Stochastic Models, Wiley-Interscience, New York.Google Scholar
  2. [2]
    Derman, C. (1970). Finite State Markovian Decision Processes, Academic Press, New York.Google Scholar
  3. [3]
    Hernandez-Lermer, O. (1989). Adaptive Markov Control Processes. Springer-Verlag, New York.Google Scholar
  4. [4]
    Howard, R. (1971). Dynamic Programming and Markov Processes, MIT Press, Cambridge, Massachusetts.Google Scholar
  5. [5]
    Puterman, M.L. (1994). Markov Decision Processes: Discrete Dynamic Programming, Wiley-Interscience, New York.Google Scholar
  6. [6]
    Ross, S.M. (1970). Applied Probability Models with Optimization Applications, Holden-Day, San Francisco.Google Scholar
  7. [7]
    Sennott, L.I. (1999). Stochastic Dynamic Programming and the Control of Queueing Systems, John Wiley, New York.Google Scholar
  8. [8]
    White, D.J. (1969). Markov Decision Processes, John Wiley, Chichester, UK.Google Scholar

Copyright information

© Kluwer Academic Publishers 2001

Authors and Affiliations

  1. 1.University of MichiganAnn ArborUSA