Markovian Decision Processes with Disturbances

Abstract

In Part II we treated two basic models of Stochastic Dynamic Programming: Control Models with independent disturbances and Markovian Decision Processes. In the first of these models the state ζ _t = ζ _{t π} in period t and the disturbance η _t+1 in the same period are stochastically independent of each other. However, there are other important models in which η _t+1 depends on ζ _t . This holds in particular in some of the Markov renewal programs treated in Chap. 22 below, where a random time η _t+1 elapses between the t-th and the (t + 1)-st decision; another relevant model is the Markovian control model MCM introduced below. We now present a model, called an MDP with disturbances, which due to a very flexible transition law comprises all these models. As a further important generalization, necessary for the Markov renewal programs, we allow the discount factor in each period to be a function of the disturbance.