Average criteria in denumerable semi-Markov decision chains under risk-aversion

Abstract

This note concerns with semi-Markov decision chains evolving on a denumerable state space. The system is directed by a risk-averse controller with constant risk-sensitivity, and the performance of a decision policy is measured by a long-run average criterion associated with bounded holding cost rates and one-step cost function. Under mild conditions on the sojourn times and the transition law, restrictions on the cost structure are given to ensure that the optimal average cost can be characterized via a bounded solution of the optimality equation. Such a result is used to establish a general characterization of the optimal average cost in terms of an optimality inequality from which an optimal stationary policy can be derived.

Appendix

Proof of Lemma 5.2

Let $(x, a)\in \mathbb {K}$ be arbitrary and notice that

$$\begin{aligned}{} & {} E\left. \left[ e^{\lambda D(X_0, A_0, X_1, S_1) - \lambda g_k S_1 + h_k(X_1)}\right| X_0 = x, A_0 = a_k\right] \nonumber \\{} & {} \qquad \qquad \qquad = \sum _{y\in S} p_{x, y}(a_k)\left( e^{\lambda C(x, a_k, y) + h_k(y)} \int _0^\infty e^{\lambda \int _0 ^s \rho _{x, a_k, y}(t)\, dt - \lambda g_k s }F_{x, a_k, y}(ds)\right) \end{aligned}$$

(6.12)

by Definition 5.1

• (i) Now, let $y\in S$ and $\varepsilon > 0$ be arbitrary but fixed, and pick $b\in (0, \infty )$ such that

  $$\begin{aligned} F_{x, a, y}(\cdot )\ \text {is continuous at}\ b\ \text {and}\ F_{x, a, y}(b) \ge 1-\varepsilon . \end{aligned}$$

  (6.13)

  Combining Assumption 2.1(ii) with the tube lemma in Munkres (2014, p. 168) there exists a neighborhood of the action $a\in A(x)$, say $V(a)\subset A(x)$, such that $|\rho _{x,\tilde{a}, y}(t) -\rho _{x,a, y}(t)|\le \varepsilon$ if $t\in [0, b]$ and $\tilde{a} \in V( a )$, and in this case, if $s\in [0, b]$ then

  $$\begin{aligned} \left| \int _0^s \rho _{x,\tilde{a}, y}(t)\, dt -\tilde{g} s - \int _0^s \rho _{x,a, y}(t)\, dt- g s\right| \le (\varepsilon + |\tilde{g} - g|)s \le (\varepsilon + |\tilde{g} - g|)b, \end{aligned}$$

  for every $g,\tilde{g}\in \mathbb {R}$. Combining this last display with the inequality $|e^x - 1|\le |x|e^{|x|}$, it follows that

  $$\begin{aligned}{} & {} \left| e^{\lambda \left[ \int _0^s \rho _{x,\tilde{a}, y}(t)\, dt -\lambda \tilde{g} s - \int _0^s \rho _{x,a, y}(t)\, dt-\lambda g s\right] } - 1 \right| \\{} & {} \qquad \qquad \le \lambda b (\varepsilon + |g - \tilde{g}|) e^{\lambda b (\varepsilon + |g - \tilde{g}|)},\quad s\in [0, b], \quad \tilde{a} \in V(a). \end{aligned}$$

  Using that $a_k\rightarrow a$, select a positive integer $N^*$ such that $a_k\in V(a)$ for $k\ge N^*$, so that the above display leads to

  $$\begin{aligned}{} & {} \left| \int _0^b e^{\lambda \int _0 ^s \rho _{x, a_k, y}(t)\, dt - \lambda g_k s } F_{x, a_k, y}(ds) - \int _0^ be^{\lambda \int _0 ^s \rho _{x, a, y}(t)\, dt - \lambda g s } F_{x, a_k, y}(ds)\right| \\{} & {} \qquad = \left| \int _0^b\left[ e^{\lambda \int _0 ^s \rho _{x, a_k, y}(t)\, dt - \lambda g_k s }- e^{\lambda \int _0 ^s \rho _{x, a, y}(t)\, dt - \lambda g s }\right] F_{x, a_k, y}(ds)\right| \\{} & {} \qquad \le \lambda b^2 (\varepsilon + |g _k- g|) e^{\lambda b (\varepsilon + |g_k - g|)} ,\quad k\ge N^*. \end{aligned}$$

  On the other hand, recalling that $g, g_k\ge 0$, observe that the inequalities

  $$\begin{aligned} e^{\lambda \int _0 ^s \rho _{x, a, y}(t)\, dt - \lambda g s }, \ e^{\lambda \int _0 ^s \rho _{x, a_k, y}(t)\, dt - \lambda g_k s} \le e^{\lambda B_\rho } \end{aligned}$$

  (6.14)

  always hold, by (2.4), so that

  $$\begin{aligned}{} & {} \left| \int _b^\infty e^{\lambda \int _0 ^s \rho _{x, a_k, y}(t)\, dt - \lambda g_k s} F_{x, a_k, y}(ds) - \int _ b^\infty e^{\lambda \int _0 ^s \rho _{x, a, y}(t)\, dt - \lambda g s} F_{x, a_k, y}(ds)\right| \\{} & {} \qquad \qquad \le 2 e^{B_\rho } (1-F_{x, a_k, y}(b)). \end{aligned}$$

  Recalling that $\{F_{x, a_k, y}(\cdot )\}_{k\in \mathbb {N}}$ converges weakly to $F_{x, a, y}(\cdot )$ and that this function is continuous at b, via (5.31) the two last displays together yield that

  $$\begin{aligned}{} & {} \limsup _{k\rightarrow \infty } \left| \int _0^\infty e^{\lambda \int _0 ^s \rho _{x, a_k, y}(t)\, dt - \lambda g_k s} F_{x, a_k, y}(ds) - \int _0^ \infty e^{\lambda \int _0 ^s \rho _{x, a, y}(t)\, dt - \lambda g s} F_{x, a_k, y}(ds)\right| \\{} & {} \qquad \qquad \le \limsup _{k\rightarrow \infty } \left[ \lambda b^2 (\varepsilon + |g_k - g|) e^{\lambda b (\varepsilon + |g_k - \tilde{g}|)} + 2 e^{B_\rho } (1-F_{x, a_k, y}(b))\right] \\{} & {} \qquad \qquad = \lambda b^2 \varepsilon e^{\lambda b \varepsilon } + 2 e^{B_\rho } (1-F_{x, a, y}(b))\le \lambda b^2 \varepsilon e^{\lambda b \varepsilon } + 2 e^{B_\rho } \varepsilon ; \end{aligned}$$

  where (6.13) was used to set the second inequality. Since $\varepsilon > 0$ is arbitrary, it follows that

  $$\begin{aligned} \lim _{k\rightarrow \infty } \left| \int _0^\infty e^{\lambda \int _0 ^s \rho _{x, a_k, y}(t)\, dt - \lambda g_k s } F_{x, a_k, y}(ds) - \int _0^ \infty e^{\lambda \int _0 ^s \rho _{x, a, y}(t)\, dt - \lambda g s } F_{x, a_k, y}(ds)\right| = 0. \end{aligned}$$

  Now, observing that the continuous mapping $s\mapsto e^{\lambda \int _0 ^s \rho _{x, a, y}(t)\, dt - \lambda g s }$ is bounded on $s\in [0,\infty )$, by (2.4) and the nonnegativity of g, Assumption 2.1(iv) implies that

  $$\begin{aligned} \lim _{k\rightarrow \infty } \left| \int _0^\infty e^{\lambda \int _0 ^s \rho _{x, a, y}(t)\, dt - \lambda gs} F_{x, a_k, y}(ds) - \int _0^ \infty e^{\lambda \int _0 ^s \rho _{x, a, y}(t)\, dt - \lambda g s} F_{x, a, y}(ds)\right| = 0, \end{aligned}$$

  and these two last displays together yield that

  $$\begin{aligned} \lim _{k\rightarrow \infty } \int _0^\infty e^{\lambda \int _0 ^s \rho _{x, a_k, y}(t)\, dt - \lambda g_k s} F_{x, a_k, y}(ds) = \int _0^ \infty e^{\lambda \int _0 ^s \rho _{x, a, y}(t)\, dt - \lambda g s} F_{x, a, y}(ds). \end{aligned}$$

  (6.15)

  Combining this convergence with (5.31) and the continuity property in Assumption 2.1(ii), after taking the inferior limit as k goes to $\infty$ in both sides of (6.12) an application of Fatou’s lemma yields that

  $$\begin{aligned}{} & {} \liminf _{k \rightarrow \infty } E\left. \left[ e^{\lambda D(X_0, A_0, X_1, S_1) - \lambda g_k S_1 + h_k(X_1)}\right| X_0 = x, A_0 = a_k\right] \\{} & {} \qquad \qquad = \liminf _{k\rightarrow \infty } \sum _{y\in S} p_{x, y}(a_k)\left( e^{\lambda C(x, a_k, y) + h_k(y)} \int _0^\infty e^{\lambda \int _0 ^s \rho _{x, a_k, y}(t)\, dt - \lambda g_k s }F_{x, a_k, y}(ds)\right) \\{} & {} \qquad \qquad \ge \sum _{y\in S} \liminf _{k\rightarrow \infty } p_{x, y}(a_k)\left( e^{\lambda C(x, a_k, y) + h_k(y)} \int _0^\infty e^{\lambda \int _0 ^s \rho _{x, a_k, y}(t)\, dt - \lambda g_k s }F_{x, a_k, y}(ds)\right) \\{} & {} \qquad \qquad = \sum _{y\in S} p_{x, y}(a)\left( e^{\lambda C(x, a, y) + h(y)} \int _0^\infty e^{\lambda \int _0 ^s \rho _{x, a, y}(t)\, dt - \lambda g s }F_{x, a_k, y}(ds)\right) \\{} & {} \qquad \qquad = E\left. \left[ e^{\lambda D(X_0, A_0, X_1, S_1) - \lambda g S_1 + h(X_1)}\right| X_0 = x, A_0 = a\right] , \end{aligned}$$

  where the Definition 5.1(i) was used to set the equality.

• (ii) Note that, by Definition 5.1(i), the assertion is equivalent to part (i).

• (iii) Using (6.14) it follows that

  $$\begin{aligned} e^{\lambda C(x, a_k, y) + \lambda h_k(y)} \int _0^\infty e^{\lambda \int _0 ^s \rho _{x, a_k, y}(t)\, dt - \lambda g_k s }F_{x, a_k, y}(ds) \le e^{\lambda (\Vert C\Vert + \Vert h_k\Vert + B_\rho )} \end{aligned}$$

  always holds, whereas Assumption 2.1(ii), (5.31) and (6.15) together imply that

  $$\begin{aligned}{} & {} \lim _{k\rightarrow \infty } e^{\lambda C(x, a_k, y) + \lambda h_k(y)} \int _0^\infty e^{\lambda \int _0 ^s \rho _{x, a_k, y}(t)\, dt - \lambda g_k s }F_{x, a_k, y}(ds)\\{} & {} \qquad \qquad \qquad \qquad = e^{\lambda C(x, a, y) + \lambda h(y)} \int _0^\infty e^{\lambda \int _0 ^s \rho _{x, a, y}(t)\, dt - \lambda g s }F_{x, a, y}(ds). \end{aligned}$$

  Since $\lim _{k\rightarrow \infty } p_{x, y}(a_k) = p_{x, y}(a)$, by Asumption 2.1(ii), if $\sup _k \Vert h_k\Vert < \infty$ then the two last displays allow to use Proposition 18 in (Royden 1988, p. 270) to obtain (5.33).