On the Expected Total Reward with Unbounded Returns for Markov Decision Processes

Abstract

We consider a discrete-time Markov decision process with Borel state and action spaces. The performance criterion is to maximize a total expected utility determined by unbounded return function. It is shown the existence of optimal strategies under general conditions allowing the reward function to be unbounded both from above and below and the action sets available at each step to the decision maker to be not necessarily compact. To deal with unbounded reward functions, a new characterization for the weak convergence of probability measures is derived. Our results are illustrated by examples.

Appendix A: Balder’s Lemma

We detail here the proof of [1, Lemma 2.4], for the sake of clarity.

Lemma A.1

Let $(\mu _{n})$ be a sequence of probability measures on a metric space Y converging weakly to a probability measure $\mu$. Consider a function $u:Y\rightarrow [-\,\infty ,+\,\infty [$, bounded from above, satisfying the following conditions: for any $\epsilon >0$, there exists a closed subset $Y_{\epsilon }$ of Y such that

$$\begin{aligned} \sup _{n\in \mathbb {N}}\mu _{n}(Y\setminus Y_{\epsilon }) <\epsilon \end{aligned}$$

(A.1)

and the restriction of u on $Y_{\epsilon }$ is upper semicontinuous. Then,

$$\begin{aligned} \mathop {\overline{\lim }}_{n\rightarrow \infty } \int _{Y} u d\mu _{n} \le \int _{Y} u d\mu . \end{aligned}$$

(A.2)

Proof

Let us assume in a first step that u is also bounded from below. Let $\Vert u\Vert =\sup _{Y}|u|$ and define, for any $\epsilon >0$, the function

$$\begin{aligned} u_\epsilon =u I_{Y_\epsilon } - \Vert u\Vert I_{Y\setminus Y_\epsilon }. \end{aligned}$$

Let us show that $u_\epsilon$ is upper semicontinuous on Y. For $\beta \in \mathbb {R}$, consider the level set

$$\begin{aligned} A_\beta =\{ x\in Y : u_\epsilon (x)< \beta \}=\{x\in Y_\epsilon : u(x)< \beta \}\cup \{x\in Y\setminus Y_\epsilon : -\Vert u\Vert < \beta \}. \end{aligned}$$

Our aim is to show that $A_\beta$ is open. If $\beta \le -\Vert u\Vert$, we clearly have $A_\beta =\emptyset$ which is an open set. Otherwise, we can write

$$\begin{aligned} A_\beta =\{x\in Y_\epsilon : u_\epsilon (x)<\beta \}\cup (Y\setminus Y_\epsilon ). \end{aligned}$$

Since u is upper semicontinuous on $Y_\epsilon$, the level set $\{x\in Y_\epsilon : u_\epsilon (x) < \beta \}$ is open in $Y_\epsilon$, and so there exists an open set O of Y such that

$$\begin{aligned} \{x\in Y_\epsilon : u_\epsilon (x)<\beta \}=Y_\epsilon \cap O. \end{aligned}$$

Thus $A_\beta =(Y_\epsilon \cap O)\cup (Y\setminus Y_\epsilon )$. Let $x\in A_\beta$. If $x\in Y\setminus Y_\epsilon$, by the closedness of $Y_\epsilon$, we can find $\eta >0$ such that $B(x,\eta )\subset Y\setminus Y_\epsilon \subset A_\beta$. Otherwise, $x\in Y_\epsilon \cap O$. In this case, since $x\in O$ which is an open set, we can find $\eta '>0$ such that $B(x,\eta ')\subset O$. Then

$$\begin{aligned} B(x,\eta ')\cap A_\beta= & {} [B(x,\eta ')\cap Y_\epsilon \cap O] \cup [ B(x,\eta ')\cap (Y\setminus Y_\epsilon )]\\= & {} [B(x,\eta ')\cap Y_\epsilon ] \cup [ B(x,\eta ')\cap Y\setminus Y_\epsilon ]=B(x,\eta '). \end{aligned}$$

Thus $B(x,\eta ')\subset A_\beta$ showing that $A_\beta$ is open. This implies that $u_\epsilon$ is upper semicontinuous on Y.

Remark that

$$\begin{aligned} \sup _{n\in \mathbb {N}}\left| \int _Y u d\mu _n -\int _Y u_\epsilon d\mu _n\right| \le 2\epsilon \Vert u\Vert . \end{aligned}$$

Now, using the fact that $u_\epsilon$ is upper semicontinuous and bounded on the whole space Y,

$$\begin{aligned} \mathop {\overline{\lim }}_{n\rightarrow \infty } \int _{Y} u d\mu _{n} \le \mathop {\overline{\lim }}_{n\rightarrow \infty } \int _{Y} u_\epsilon d\mu _{n}+2\Vert u\Vert \epsilon \le \int _{Y} u_\epsilon d\mu +2\Vert u\Vert \epsilon \le \int _{Y} u d\mu +2\Vert u\Vert \epsilon , \end{aligned}$$

showing the result.

In the case where u is no longer bounded from below, we introduce $u_m=u\vee (-m)$ for which the previous step holds. Then, we apply the monotone convergence theorem to obtain the result. $\square$