Markov Decision Processes with Discounted Rewards: New Action Elimination Procedure

Abstract

Since the computational complexity is one among stations of interest of many interested researchers, numerous procedures are appeared for accelerating iterative methods and for reducing the memory bits required for computing machines. For solving Markov Decision Processes (MDPs), several tests are proposed in the literature and especially to improve the standard Value Iteration Algorithm (VIA). The Bellman optimality equation have played a central role for establish this dynamic programming tool.

In this work, we propose a new test based on the extension of some test for eliminating non-optimal decisions from the planning. In order to demonstrate the scientific interest of our contribution, we compare our result with those of Macqueen and Porteus by an illustrating example. Thus, we reduce the state and action spaces size in each stage as soon as it is possible.

Appendix

Now, we give reminders about some famous tests that have played a crucial role in action elimination approach notably in Markov decision problems. Applying MacQueen’s [1, 2], Porteus [3], Semmouri, Jourhmane and Elbaghazaoui [10] bounds for the standard VIA leads to the following tests to eliminate action a in A(s) permanently:

MacQueen test:

$$M^{n}(i,a):=Q^{n}(i,a)-V_{i}^{n}-\gamma \frac{a_{n}-b_{n}}{1-\gamma }\prec 0$$

Porteus test:

$$P^{n}(i,a):=Q^{n}(i,a)-V_{i}^{n}-\gamma ^{2} \frac{a_{n-1}-b_{n-1}}{1-\gamma }\prec 0$$

where

$$a_{n}=\min _{i\in S}(V_{i}^{n}- V_{i}^{n-1}), \,\, b_{n}=\max _{i\in S}(V_{i}^{n}- V_{i}^{n-1})$$

and

$$Q^{n}=r(i,a)+\gamma \sum _{j\in S}p(j/,i,a)V_{j}^{n}$$

Let also

$$\begin{aligned} n_{\gamma }(M)&=\inf \{n\in \mathbb {N}/\exists i\in S,\exists a\in A(i), \, M^{n}(i,a)\prec 0\}\\ N_{\gamma }(M)&=\sup \{n\in \mathbb {N}/\exists i\in S,\exists a\in A(i), \, M^{n}(i,a)\prec 0\} \end{aligned}$$

and

$$\begin{aligned} n_{\gamma }(P)&=\inf \{n\in \mathbb {N}/\exists i\in S,\exists a\in A(i), \, P^{n}(i,a)\prec 0\}\\ N_{\gamma }(P)&=\sup \{n\in \mathbb {N}/\exists i\in S,\exists a\in A(i), \, P^{n}(i,a)\prec 0\} \end{aligned}$$

Semmouri, Jourhmane and Elbaghazaoui test:

$$SJE^{n}(i,a):=V_{i}^{n}-Q^{n}(i,a)+\frac{(1+\gamma )C}{1-\gamma }\gamma ^{n}\prec 0$$

where

$$C=\max _{i,a}|c(i,a)|$$

and let too

$$\begin{aligned} n_{\gamma }(SJE1)&=\inf \{n\in \mathbb {N}/\exists i\in S,\exists a\in A(i), \, SJE^{n}(i,a)\prec 0\}\\ N_{\gamma }(SJE1)&=\sup \{n\in \mathbb {N}/\exists i\in S,\exists a\in A(i), \, SJE^{n}(i,a)\prec 0\} \end{aligned}$$