Illustrated review of convergence conditions of the value iteration algorithm and the rolling horizon procedure for average-cost MDPs

Abstract

This paper is concerned with the links between the Value Iteration algorithm and the Rolling Horizon procedure, for solving problems of stochastic optimal control under the long-run average criterion, in Markov Decision Processes with finite state and action spaces. We review conditions of the literature which imply the geometric convergence of Value Iteration to the optimal value. Aperiodicity is an essential prerequisite for convergence. We prove that the convergence of Value Iteration generally implies that of Rolling Horizon. We also present a modified Rolling Horizon procedure that can be applied to models without analyzing periodicity, and discuss the impact of this transformation on convergence. We illustrate with numerous examples the different results. Finally, we discuss rules for stopping Value Iteration or finding the length of a Rolling Horizon. We provide an example which demonstrates the difficulty of the question, disproving in particular a conjectured rule proposed by Puterman.

Appendix

Each month an individual must decide how to allocate his wealth between different consumptions and investments. Each state represents a level of individual’s wealth at the start of a month. Wealth levels give access to two different investment opportunities, prudent or risky. Choosing an investment profile at each level results in a probability transition for the next wealth level, as well as an instantaneous gain. The individual’s objective is to maximize the average gain.

There are five levels of wealth, ordered from the smallest to the largest. At the medium level, connected to the risky behavior, there exists positive probability to pass to the next inferior level of wealth. It is also possible to cycle among the two inferior levels, but there is no action which permit the access to the three superior levels from the inferior ones. Besides, being at the poorest level, by some external help we achieve level 2. There is a common action space A={a ₁,a ₂}, where a ₁ represents the prudent investment profile and a ₂ the risky attitude. We show the data below. $P_{a_{k}}(s,j)$ is the transition probability from the state s to state j when action a _k is used, i.e. $P_{a_{k}}(s,j)=p(j|s,a_{k})$.

The gains can summarize as follows: r(s,a _k ) in the matrix below is the gain when at state s, the action a _k is chosen.

$$\left(\begin{array}{c@{\quad}c}1 & 2\\1 & 2\\1 & 1\\3 & 2\\6 & 6\end{array}\right).$$

Through the implementation of the MRH procedure the optimal average wealth can be computed: g ^∗=(2,2,4,4,4). It is produced by the stationary policy associated to the decision rule d=(a ₂,a ₂,a ₁,a ₂,a ₁) whose transition matrix is:

$$\left (\begin{array}{c@{\quad}c@{\quad}c@{\quad}c@{\quad}c}0 & 1 & 0 & 0 & 0\\1& 0 & 0 & 0& 0\\0& 0& 0.7 & 0.3& 0 \\0& 0& 0& 0 & 1\\0& 0& 0& 1 & 0\end{array} \right).$$

Clearly, it is a multichain periodic model. When RH procedure is applied directly, there is no convergence: the procedure gives infinitely (and periodically) many times two policies, (a ₂,a ₂,a ₁,a ₁,a ₁) and (a ₂,a ₂,a ₁,a ₂,a ₁). The first one produces a gain g=(2,2,3,3,3) and then it is not optimal.