Integro-differential optimality equations for the risk-sensitive control of piecewise deterministic Markov processes

Abstract

In this paper we study the minimization problem of the infinite-horizon expected exponential utility total cost for continuous-time piecewise deterministic Markov processes with the control acting continuously on the jump intensity $\lambda$ and on the transition measure Q of the process. The action space is supposed to depend on the state variable and the state space is considered to have a frontier such that the process jumps whenever it touches this boundary. We characterize the optimal value function as the minimal solution of an integro-differential optimality equation satisfying some boundary conditions, as well as the existence of a deterministic stationary optimal policy. These results are obtained by using the so-called policy iteration algorithm, under some continuity and compactness assumptions on the parameters of the problem, as well as some non-explosive conditions for the process.

Appendix

Proof of Proposition 4.1

Clearly (30) holds for $x^{\infty }$. For $x\in \mathbf {X}$, the proof follows similar steps as the proof of Proposition 3.10 in Costa and Dufour (2013). Consider $\Theta _n=\bigl (\pi _n,\gamma _{n}\bigr )\in \mathbb {V}^{r}(x)$ and $\Theta =\bigl (\pi ,\gamma \bigr )\in \mathbb {V}^{r}(x)$ such that $\Theta _n \rightarrow \Theta$ [see (2)], and a sequence of functions $h_{n}\in \mathbb {M}_\infty (\mathbf {X}_{\infty })$ greater than 1, and set $h = \mathop {\underline{\lim }}_{n\rightarrow \infty } h_{n}$. From Assumption D and E we can find a non-decreasing sequence of functions $C^\ell (x,.)$, (${\widetilde{C}}^\ell (z,.)$ respectively), $\ell \in \mathbb {N}$, converging to $C^g(x,.)$ such that $C^\ell (x,a)$ is continuous for $a\in \mathbf {A}(x)$ (converging to $C^i(z,.)$ such that ${\widetilde{C}}^\ell (z,a)$ is continuous for $a\in \mathbf {A}(z)$). Set for $\rho \in \mathbb {M}(\mathbf {K})$, $\rho (x,a) \ge 0$, $g\in \mathbb {M}_\infty (\mathbf {X})$, $g\ge 1$, $\ell \in \mathbb {N}$, and $t\in [0,t^*(x)]$,

$$\begin{aligned}&P_n^\ell (g,t) = \int _{]0,t[} \exp \Big [ \int _{]0,s[}\Big (C^{\ell }_{\pi _n} \big (\phi (x,r)\big )dr- \lambda _{\pi _n} \big (\phi (x,r)\big )\Big ) dr \Big ] g(\phi (x,s)\big ) ds,\\&P^\ell (g,t) = \int _{]0,t[} \exp \Big [ \int _{]0,s[}\Big (C^{\ell }_{\pi } \big (\phi (x,r)\big )dr- \lambda _{\pi } \big (\phi (x,r)\big )\Big ) dr \Big ] g(\phi (x,s)\big )ds, \end{aligned}$$

and

$$\begin{aligned} T_n^\ell (\rho ,g,t)&= P_n^\ell (\rho Q_{\pi _n} g,t)\\&= \int _{]0,t[} \exp \Big [ \int _{]0,s[}\Big (C^{\ell }_{\pi _n} \big (\phi (x,r)\big )dr- \lambda _{\pi _n} \big (\phi (x,r)\big )\Big ) dr \Big ]\rho Q_{\pi _n}g(\phi (x,s)\big ) ds,\\ {\widetilde{T}}_n^\ell (\rho ,g,t)&= P^\ell (\rho Q_{\pi _n} g,t)\\&= \int _{]0,t[} \exp \Big [ \int _{]0,s[}\Big (C^{\ell }_{\pi } \big (\phi (x,r)\big )dr- \lambda _{\pi } \big (\phi (x,r)\big )\Big ) dr \Big ]\rho Q_{\pi _n}g(\phi (x,s)\big ) ds,\\ S_n(g)&= \exp \Big [ \int _{]0,t^{*}(x)[} C^{g}_{\pi _n} \big (\phi (x,s)\big ) ds +C^{i}_{\gamma _n}(\phi (x,t^{*}(x)))\Big ]\\&\quad \exp \Big [ - \int _{]0,t^{*}(x)[} \lambda _{\pi _n} \big (\phi (x,s)\big )ds \Big ] \\&\quad Q_{\gamma _n}g(\phi (x,t^{*}(x))\big ),\\ T^\ell (\rho ,g,t)&= P^\ell (\rho Q_{\pi } g,t)\\&= \int _{]0,t[} \exp \Big [ \int _{]0,s[}\Big (C^{\ell }_{\pi } \big (\phi (x,r)\big )dr- \lambda _{\pi } \big (\phi (x,r)\big )\Big ) dr \Big ] \rho Q_{\pi }g(\phi (x,s)\big ) ds,\\ S(g)&= \exp \Big [ \int _{]0,t^{*}(x)[} C^{g}_{\pi } \big (\phi (x,s)\big )ds +C^{i}_{\gamma }(\phi (x,t^{*}(x)))\Big ]\\&\quad \exp \Big [ - \int _{]0,t^{*}(x)[} \lambda _{\pi } \big (\phi (x,s)\big )\Big )ds \Big ] \\&\quad Q_{\gamma }g(\phi (x,t^{*}(x))\big ). \end{aligned}$$

We also define the functions $P_n(g,t)$, P(g, t), $T_n(\rho ,g,t)$, ${\widetilde{T}}_n(\rho ,g,t)$, $T(\rho ,g,t)$, replacing $C^\ell$ by $C^g$. Notice that

$$\begin{aligned} T_n^\ell (\rho ,g,t) = P_n^\ell (\rho Q_{\pi _n} g,t) - P^\ell (\rho Q_{\pi _n} g,t) + {\widetilde{T}}_n^\ell (\rho ,g,t), \end{aligned}$$

(62)

and that

$$\begin{aligned}&\mathcal {T}h_{n}(\Theta _n,x) = T_n(\lambda ,h_n,t^{*}(x)) + S_n(h_n), \end{aligned}$$

(63)

$$\begin{aligned}&\mathcal {T}h(\Theta ,x) = T(\lambda ,h,t^{*}(x)) + S(h). \end{aligned}$$

(64)

Our goal will be to show that

$$\begin{aligned} \mathop {\underline{\lim }}_{n\rightarrow \infty } {T}_n(\lambda ,h_n,t^*(x)) \ge&T(\lambda ,h,t^*(x)), \end{aligned}$$

(65)

and that

$$\begin{aligned} \mathop {\underline{\lim }}_{n\rightarrow \infty } S_n(h_n) \ge S(h). \end{aligned}$$

(66)

From (63), (64), (65) and (66) we get that

$$\begin{aligned}&\mathop {\underline{\lim }}_{n\rightarrow \infty }\mathcal {T}h_{n}(\Theta _n,x) \\&\quad = \mathop {\underline{\lim }}_{n\rightarrow \infty }(T_n(\lambda ,h_n,t^{*}(x)) + S_n(h_n)) \ge \mathop {\underline{\lim }}_{n\rightarrow \infty } T_n(\lambda ,h_n,t^{*}(x)) + \mathop {\underline{\lim }}_{n\rightarrow \infty } S_n(h_n) \\&\quad \ge T(\lambda ,h,t^{*}(x)) + S(h) = \mathcal {T}(h)(\Theta ,x) \end{aligned}$$

which will prove our result.

We will first show that (65) holds. Following the same reasoning as in the proof of item (a) of Proposition 3.10 in Costa and Dufour (2013) (page 38), and using Assumptions B, C and D we get that for any $t\in [0,t^*(x)] \cap \mathbb {R}_+$,

$$\begin{aligned}&\lim _{n\rightarrow \infty } \int _{0}^t \lambda _{\pi _n}(\phi (x,s))ds \nonumber \\&\quad = \int _{0}^t \lambda _\pi (\phi (x,s)) ds, \,\, \lim _{n\rightarrow \infty } \int _{0}^t C_{\pi _n}^\ell (\phi (x,s)) ds= \int _{0}^t C_\pi ^\ell (\phi (x,s)) ds. \end{aligned}$$

(67)

Set for any $s\in [0,t* (x)]\cap \mathbb {R}_+$,

$$\begin{aligned} \Psi _n^\ell (s)&=\exp \Big [ \int _{]0,s[}\Big (C^{\ell }_{\pi _n} \big (\phi (x,r)\big )- \lambda _{\pi _n} \big (\phi (x,r)\big )\Big ) dr \Big ] \\&\quad - \exp \Big [ \int _{]0,s[}\Big (C^{\ell }_{\pi } \big (\phi (x,r)\big )- \lambda _{\pi } \big (\phi (x,r)\big )\Big )\Big ] dr. \end{aligned}$$

From (67) we have that

$$\begin{aligned} \lim _{n\rightarrow \infty } \Psi _n^\ell (s)= 0, \end{aligned}$$

(68)

and, moreover, from Assumption G, that for any $s<t^{*}(x)$

$$\begin{aligned} | \Psi _n^\ell (s) | \le 2\exp \Big [ \int _{]0,s[}\Big (\overline{C}\big (\phi (x,r)\big )dr- \underline{\lambda } \big (\phi (x,r)\big )\Big ) dr \Big ] \end{aligned}$$

and, from (29), that for any $t<t^{*}(x)$

$$\begin{aligned} \int _{]0,t[} \exp \Big [ \int _{]0,s[}\Big (\overline{C}\big (\phi (x,r)\big )dr- \underline{\lambda } \big (\phi (x,r)\big )\Big ) dr \Big ] ds < \infty . \end{aligned}$$

From this, (68) and the bounded convergence theorem we conclude that

$$\begin{aligned} \lim _{n\rightarrow \infty } \int _{]0,t[}| \Psi _n^\ell (s) | ds = \int _{]0,t[} \lim _{n\rightarrow \infty } | \Psi _n^\ell (s) | ds = 0. \end{aligned}$$

(69)

Define ${\underline{h}}_k = \inf _{j\ge k} h_j$ so that ${\underline{h}}_k\le h_n$ for $n\ge k$, and ${\underline{h}}_k \uparrow h$. Set $\lambda ^m(y,a) = \lambda (y,a)\wedge m$, ${\underline{h}}_k^m(y,a) ={\underline{h}}_k(y,a)\wedge m$. We have from (62) that

$$\begin{aligned} T_n^\ell (\lambda ^m,{\underline{h}}_k^m,t) = P_n^\ell (\lambda ^m Q_{\pi _n} {\underline{h}}_k^m,t) - P^\ell (\lambda ^m Q_{\pi _n} {\underline{h}}_k^m,t) + {\widetilde{T}}_n^\ell (\lambda ^m,{\underline{h}}_k^m,t). \end{aligned}$$

(70)

Since $0\le \lambda ^m Q_{\pi _n} {\underline{h}}_k^m (\phi (x,s)) \le m^2$ we get from (69) that

$$\begin{aligned}&\lim _{n\rightarrow \infty } |P_n^\ell (\lambda ^m Q_{\pi _n} {\underline{h}}_k^m,t) - P^\ell (\lambda ^m Q_{\pi _n} {\underline{h}}_k^m,t)|\nonumber \\&\quad = \lim _{n\rightarrow \infty }\Big | \int _{]0,t[} \Psi _n^\ell (s) \lambda ^m Q_{\pi _n} {\underline{h}}_k^m(\phi (x,s)\big ) ds \Big | \nonumber \\&\quad \le \lim _{n\rightarrow \infty }\int _{]0,t[} | \Psi _n^\ell (s) | ds \,m^2 = 0. \end{aligned}$$

(71)

From Assumptions C and F we have that for each m, $\lambda _m Q {\underline{h}}_k^m \in \mathbb {B}(\mathbf {K})$, it is positive, and for any $y\in E$, $\lambda _m Q {\underline{h}}_k^m (y,.)$ is continuous on $\mathbf {A}(y)$. Therefore, from the fact that $\Theta _n \rightarrow \Theta$, we get that [see (2)]

$$\begin{aligned}&\lim _{n\rightarrow \infty } {\widetilde{T}}_n^\ell (\lambda ^m,{\underline{h}}_k^m,t)\nonumber \\&\quad = \lim _{n\rightarrow \infty } \int _{]0,t[} \int _{\mathbf {A}(\phi (x,t))} \exp \Big [ \int _{]0,s[}\Big (C^{\ell }_{\pi } \big (\phi (x,r)\big )dr- \lambda _{\pi } \big (\phi (x,r)\big )\Big ) dr \Big ] \nonumber \\&\times \Big [ \lambda ^m(\phi (x,s),a) Q{\underline{h}}_k^m (\phi (x,s),a)\Big ]\pi _{n}(da|\phi (x,s)) ds \nonumber \\&\quad = \int _{]0,t[} \int _{\mathbf {A}(\phi (x,t))} \exp \Big [ \int _{]0,s[}\Big (C^{g}_{\pi } \big (\phi (x,r)\big )dr- \lambda _{\pi } \big (\phi (x,r)\big )\Big ) dr \Big ] \nonumber \\&\qquad \times \Big [ \lambda ^m(\phi (x,s),a) Q{\underline{h}}_k^m(\phi (x,s),a)\Big ]\pi (da|\phi (x,s)) ds = T(\lambda ^m,{\underline{h}}_k^m,t), \end{aligned}$$

(72)

so that from (70), (71), (72), we have that

$$\begin{aligned} \lim _{n\rightarrow \infty } T_n^\ell (\lambda ^m,{\underline{h}}_k^m,t) = \lim _{n\rightarrow \infty } {\widetilde{T}}_n^\ell (\lambda ^m,{\underline{h}}_k^m,t)=&T^\ell (\lambda ^m,{\underline{h}}_k^m,t). \end{aligned}$$

(73)

Since $C^g \ge C^\ell$ for $\ell \in \mathbb {N}$, $h_n \ge {\underline{h}}_k \ge {\underline{h}}_k^m$ for $n\ge k$, $\lambda \ge \lambda ^m$, we get from (73) that

$$\begin{aligned} \mathop {\underline{\lim }}_{n\rightarrow \infty } {T}_n(\lambda ,h_n,t) \ge \lim _{n\rightarrow \infty } T_n^\ell (\lambda ^m,{\underline{h}}_k^m,t) \ge T^\ell (\lambda ^m,{\underline{h}}_k^m,t). \end{aligned}$$

(74)

From the monotone convergence theorem and taking the limit over $\ell$, m and k we get from (74) that

$$\begin{aligned} \mathop {\underline{\lim }}_{n\rightarrow \infty } T_n(\lambda ,h_n,t) \ge&T(\lambda ,h,t). \end{aligned}$$

(75)

Since $\lambda$ and $h_n$, h are positive, we have that $T_n(\lambda ,h_n,t^*(x)) \ge T_n(\lambda ,h_n,t)$, so that from (75),

$$\begin{aligned} \mathop {\underline{\lim }}_{n\rightarrow \infty } {T}_n(\lambda ,h_n,t^*(x))\ge \mathop {\underline{\lim }}_{n\rightarrow \infty } {T}_n(\lambda ,h_n,t) \ge&T(\lambda ,h,t). \end{aligned}$$

(76)

Taking the limit as $t \rightarrow t^*(x)$ we conclude from (76) that (65) holds.

Let us show now that (66) holds. We consider two cases, the first one is for $t^*(x)=\infty$, and the second one is for $t^*(x)<\infty$. For the case $t^*(x)=\infty$ we recall first that $Q_{\gamma _n}h_n(\phi (x,\infty )) = Q_{\gamma _n}h_n(x^{\infty })=1$, $Q_{\gamma }h(\phi (x,\infty )) = Q_{\gamma }h(x^{\infty })=1$, and that $C^i_{\gamma _n}(\phi (x,\infty )) = C^i_{\gamma _n}(x^{\infty })=0$, $C^i_{\gamma }(\phi (x,\infty )) = C^i_{\gamma }(x^{\infty })=0$. Suppose that $\int _{]0,\infty [}\lambda _{\pi } \big (\phi (x,r)\big ) dr=\infty$. In this case, we have that $S(h) = 0$ recalling that, by convention, $0 \times \infty =0$. Therefore, $\mathop {\underline{\lim }}_{n\rightarrow \infty } S_n(h_n) \ge S(h)$.

For the case $\int _{]0,\infty [}\lambda _{\pi } \big (\phi (x,r)\big ) dr<\infty$ we have that $\int _{]0,\infty [} \underline{\lambda }(\phi (x,s))ds \le \int _{]0,\infty [}\lambda _{\pi } \big (\phi (x,r)\big ) dr < \infty$ and from Assumption G, $\int _{]0,\infty [} \overline{\lambda }(\phi (x,s))ds < \infty$. Thus $\lambda (\phi (x,.),.)\in L^{1}(\mathbb {R}_{+};\mathbb {C}(\mathbf {A}))$ and from (2) we have that $\lim _{n\rightarrow \infty } \int _{]0,\infty [}\lambda _{\pi _n} \big (\phi (x,r)\big ) dr = \int _{]0,\infty [}\lambda _{\pi } \big (\phi (x,r)\big ) dr$. Therefore,

$$\begin{aligned} \mathop {\underline{\lim }}_{n\rightarrow \infty } S_n(h_n)&\ge \lim _{n\rightarrow \infty }\Big \{\exp \Big [ - \int _{]0,\infty [}\lambda _{\pi _n} \big (\phi (x,r)\big ) dr \Big ] \exp \Big [ \int _{]0,t[}C_{\pi _n}^\ell \big (\phi (x,r)\big ) dr \Big ]\Big \}\\&= \exp \Big [ - \int _{]0,\infty [}\lambda _{\pi } \big (\phi (x,r)\big ) dr \Big ] \exp \Big [ \int _{]0,t[}C_{\pi }^\ell \big (\phi (x,r)\big ) dr \Big ] . \end{aligned}$$

Taking the limit as $\ell \rightarrow \infty$ and monotone convergence theorem we get that

$$\begin{aligned} \mathop {\underline{\lim }}_{n\rightarrow \infty } S_n(h_n)&\ge \exp \Big [ - \int _{]0,\infty [}\lambda _{\pi } \big (\phi (x,r)\big ) dr \Big ] \exp \Big [ \int _{]0,t[}C_{\pi } \big (\phi (x,r)\big ) dr \Big ] \end{aligned}$$

and by taking the limit as $t\rightarrow \infty$ we conclude that $\mathop {\underline{\lim }}_{n\rightarrow \infty } S_n(h_n) \ge S(h)$. Consider now the case $t^*(x)<\infty$. From Assumption E we get that

$$\begin{aligned} \lim _{n\rightarrow \infty } \exp \Big [ {\widetilde{C}}^{\ell }_{\gamma _n}(\phi (x,t^{*}(x))) \Big ] = \exp \Big [ {\widetilde{C}}^{\ell }_{\gamma }(\phi (x,t^{*}(x))) \Big ]. \end{aligned}$$

(77)

Defining again ${\underline{h}}_k = \inf _{j\ge k} h_j$, ${\underline{h}}_k^m = {\underline{h}}_k \wedge m$, and repeating similar arguments as above we get that

$$\begin{aligned}&\mathop {\underline{\lim }}_{n\rightarrow \infty } S_n(h_n)\\&\quad \ge \lim _{n\rightarrow \infty } \exp \Big [ \int _{]0,t^{*}(x)[} \Big ( C^{\ell }_{\pi _n} \big (\phi (x,s)\big )-\lambda _{\pi _n} \big (\phi (x,s)\big )\Big )ds\\&\qquad +{\widetilde{C}}^{\ell }_{\gamma _n}(\phi (x,t^{*}(x))) \Big ] Q_{\gamma _n} {\underline{h}}_k^m (\phi (x,t^{*}(x)))\\&\quad = \exp \Big [ \int _{]0,t^{*}(x)[} \Big ( C^{\ell }_{\pi } \big (\phi (x,s)\big )-\lambda _{\pi } \big (\phi (x,s)\big )\Big )ds\\&\qquad +{\widetilde{C}}^{\ell }_{\gamma }(\phi (x,t^{*}(x))) \Big ] Q_{\gamma }{\underline{h}}_k^m (\phi (x,t^{*}(x))). \end{aligned}$$

Taking the limit as $\ell$, m, k goes to $\infty$ and from the monotone convergence theorem we get that $\mathop {\underline{\lim }}_{n\rightarrow \infty } S_n(h_n) \ge S(h)$ and thus we conclude that (66) hods. $\square$