Markov Decision Processes

Abstract

In this chapter, we add an extra layer of complexity to our models of decision making by introducing the idea of a state-dependent decision process. The processes we will consider can either be deterministic or stochastic. To begin with, we will assume that the process must terminate by an a priori fixed time T (a “finite horizon” model). In principle, decisions can be made at times t = 0, 1, 2,..., T — 1, although the actual number of decisions made may be fewer than T if the process terminates early as a consequence of the actions taken. Models that have no a priori restriction on the number of decisions to be taken (“infinite horizon” models) will be considered in the next chapter.