Linear-Quadratic Stochastic Delayed Control and Deep Learning Resolution

Abstract

We consider a simple class of stochastic control problems with a delayed control, in both the drift and the diffusion part of the state stochastic differential equation. We provide a new characterization of the solution in terms of a set of Riccati partial differential equations. Existence and uniqueness of a solution are obtained under a sufficient condition expressed directly as a relation between the time horizon, the drift, the volatility and the delay. Furthermore, a deep learning scheme (The code is available in a IPython notebook.) is designed and used to illustrate the effect of the delay feature on the Markowitz portfolio allocation problem with execution delay.

A Proof of Proposition 3.3

Our proof extends [1, see Theorem 5] to the case where the volatility is controlled. It consists in slicing the domain ${\mathcal {D}}$ in slices of size d and proceeding by a backward recursion. More precisely, we show existence and uniqueness over a sequence of slices $\left( [T-(n+1)d, T-nd] \times [-d, 0]^2\right) _{n}$. We then concatenate the sequence of absolutely continuous solutions obtained, which yields a piecewise absolutely continuous solution. In each slice, the proof consists of the following steps

• (1) Show that there exists a unique solution on a small interval;

• (2) Prove that the local solution is Lipschitz;

• (3) As a result extend the solution to the whole slice.

We finally concatenate the sequence of solutions obtained above.

1.1 A.1 Slice $t \in [T-d, T]$, Initialization

On ${\mathcal {D}}_b \cup {\mathcal {D}}_c$, the constraints (2.9)–(2.10)–(2.11) on $P_{12}, P_{\hat{22}}$ and $P_{22}$ reduce to linear homogeneous transport equations admitting closed form solutions given, for every $(t,s,r)\in {\mathcal {D}}_{b} \cup {\mathcal {D}}_c$, by

$$\begin{aligned}&P_{12}(t,s) = b P_{11}(t+s+d)1_{t+s+d \le T}, \qquad P_{\hat{22}}(t,s) = \sigma ^2 P_{11}(t+s+d)1_{t+s+d \le T}, \\&P_{22}(t,s,r) = b^2 P_{11}(t+ s \vee r + d)1_{t+s\vee r+d \le T}. \end{aligned}$$

Or, as $P_{11}(t) = 1$ for any $t \ge T-d$, we then have for every $(t,s,r)\in {\mathcal {D}}_{b} \cup {\mathcal {D}}_c$

$$\begin{aligned}&P_{11}(t) = 1, \qquad P_{12}(t,s) = b1_{t+s+d \le T}, \\&P_{\hat{22}}(t,s) = \sigma ^21_{t+s+d \le T}, \qquad P_{22}(t,s,r) = b^21_{t+s\vee r+d \le T}. \end{aligned}$$

The existence and uniqueness in the sense of Definition 2.6 are thus trivially proved on $[T-d, T]$.

1.2 A.2 Slice $[T - 2d, T- d]$

On $[T-2d, T-d]\times [-d, 0]^2$, we have $P_{\hat{22}}(t,s) = \sigma ^2 P_{11}(t+s+d)$ so that $P_{\hat{22}}(t,0) = \sigma ^2 P_{11}(t+d)=\sigma ^2$. Consequently, the system (2.9)–(2.10)–(2.11) reduces to

$$\begin{aligned}&\dot{P}_{11}(t) =\frac{P_{12}(t,0)^{2}}{\sigma ^2}, \nonumber \\&(\partial _t - \partial _s)(P_{12})(t,s) =\frac{P_{12}(t,0)P_{22}(t,s,0)}{\sigma ^2}, \nonumber \\&(\partial _t - \partial _s-\partial _r)(P_{22})(t,s,r) =\frac{P_{22}(t,s,0)P_{22}(t,0,r)}{\sigma ^2}, \end{aligned}$$

(A.1)

with terminal conditions

$$\begin{aligned} P_{11}(T-d) =1, \quad P_{12}(T-d, s) = b, \quad P_{22}(T-d, s, r) = b^2, \end{aligned}$$

(A.2)

and boundary constraints

$$\begin{aligned}&P_{12}(t,-d) = b P_{11}(t), \quad P_{22}(t,s,-d) = b P_{12}(t,s). \end{aligned}$$

(A.3)

Thus, for every $(t,s,r)\in [T-2d, T-d]\times [-d,0]^2$, the set of equations (A.1) and constraints (A.2)–(A.3) can be rewritten in the following integral form

$$\begin{aligned} P_{11}(t)&= 1- \sigma ^{-2}\int _{t}^{T-d} {P_{12}(x,0)^{2}}\mathrm{d}x,\nonumber \\ P_{12}(t,s)&= b P_{11}((T-d) \wedge (t+s+d)) \nonumber \\&\quad - \sigma ^{-2} \int _{t}^{(T-d) \wedge (t+s+d)} {P_{12}(x,0)P_{22}(x,t+s-x,0)}\mathrm{d}x,\nonumber \\ P_{22}(t,s,r)&= b P_{12}((T-d) \wedge (t+s \wedge r+d),(s-r) \vee (r-s)-d)\nonumber \\&\quad - \sigma ^{-2} \int _{t}^{(T-d) \wedge (t+s \wedge r+d)} {P_{22}(x,t+s-x,0) P_{22}(x,0,t+r-x)}\mathrm{d}x. \end{aligned}$$

(A.4)

We then make use of the following lemma to prove local existence of a solution.

Lemma A.1

There exists $\tau \in (0, d]$ such that system (A.4) has a unique absolutely continuous solution on $[T-\tau -d, T-d]\times [-d, 0]^2$.

Proof

Let $\tau \in (0, d]$ and ${\mathcal {S}}_\tau$ denote the Banach space of absolutely continuous functions $\xi =\left( \xi _{1}(\cdot ),\xi _{2} (\cdot ,\cdot ),\xi _{3}(\cdot ,\cdot ,\cdot )\right)$ defined on

$$\begin{aligned} {\mathcal {D}}_{\tau }=\left\{ (t,s,r)|\ T-d-\tau \le t\le T-d, -d\le s,r\le 0 \right\} , \end{aligned}$$

endowed with the sup-norm

$$\begin{aligned} \Vert \xi \Vert _\infty =\Vert \xi _1\Vert _\infty + \Vert \xi _2\Vert _\infty + \Vert \xi _3\Vert _\infty , \end{aligned}$$

where $\Vert \xi _1\Vert _\infty , \Vert \xi _2\Vert _\infty$ and $\Vert \xi _3\Vert _\infty$ denote, with a slight abuse of notation, the respective sup-norm on $[T-d-\tau , T-d]$, $[T-d-\tau , T-d]\times [-d, 0]$ and $[T-d-\tau , T-d]\times [-d, 0]^2$. Let ${\mathcal {B}}_\tau$ denote the ball in ${\mathcal {S}}_\tau$

$$\begin{aligned} {\mathcal {B}}_\tau = \{ (\xi _1, \xi _2, \xi _3) \in {\mathcal {S}}_\tau : \quad \Vert \xi _1-1\Vert _{{ \infty }} \le 1/2, \quad \Vert \xi _2 -b\Vert _{{ \infty }} \le |b|/2, \quad \Vert \xi _3 - b^2\Vert _{{ \infty }} \le b^2/2 \}, \end{aligned}$$

On ${\mathcal {B}}_\tau$, we denote by $\phi =\left( \phi _{1},\phi _{2},\phi _{3}\right)$ the operator defined as follows

$$\begin{aligned} \left( \phi _{1}\xi \right) (t)&= 1 - \sigma ^{-2}\int _{t}^{T-d}{\xi _{2}(x,0)^{2}}\mathrm{d}x\\ \left( \phi _{2}\xi \right) (t,s)&= b \phi _1(\xi )((T-d) \wedge (t+s+d)) \\&\quad -\sigma ^{-2}\int _{t}^{(T-d) \wedge (t+s+d)} {\xi _{2}(r,0)\xi _{3}(r,t+s-x,0)}\mathrm{d}x\\ \left( \phi _{3}\xi \right) (t,s,r)&= {b\phi _2(\xi ) \left( (T-d) \wedge (t+s \wedge r+d),(s-r) \vee (r-s)-d \right) }\\&\quad -\sigma ^{-2}\int _{t}^{(T-d) \wedge (t+s \wedge r+d)} {\xi _{3} (x,t+s-x,0)\xi _{3}(x,0,t+r-x)} \mathrm{d}x. \end{aligned}$$

Clearly, there exists $\tilde{\tau }>0$ such that for any $\tau \le \tilde{\tau }$, $\phi ({\mathcal {B}}_{ \tau }) \rightarrow {\mathcal {B}}_{ \tau }$. We show a contraction property on $\phi$. For any $\xi , \xi ' \in {\mathcal {B}}_{\tau }$, we have the following inequalities

$$\begin{aligned} \Vert \phi _{1}(\xi )-\phi _{1}(\xi ')\Vert _\infty&\le 4\tau \sigma ^{-2} |b| \Vert \xi _2 - \xi _2' \Vert _\infty ,\\ \Vert \phi _{2}(\xi )- \phi _{2}(\xi ') \Vert _{\infty }&\le 4\tau \sigma ^{-2} \left( |b| \Vert \xi _2-\xi _2' \Vert _\infty + |b|^2 \Vert \xi _3-\xi _3' \Vert _\infty \right) \\&\quad + |b| \Vert \phi _1(\xi ) - \phi _1( \xi ')\Vert _\infty ,\\ \Vert \phi _{3}(\xi ) - \phi _{3}(\xi ')\Vert _\infty&\le |b| \Vert \phi _2(\xi ) -\phi _2(\xi ')\Vert _\infty + 4\tau \sigma ^{-2} |b|^2 \Vert \xi _3-\xi _3'\Vert _\infty . \end{aligned}$$

Consequently, the operator $\phi$ satisfies

$$\begin{aligned} \Vert \phi (\xi ) - \phi (\xi ')\Vert _\infty \le \tau m \Vert \xi -\xi '\Vert _\infty , \end{aligned}$$

where $m>0$ depends on b and $\sigma$. Therefore, for $\tau < \tilde{\tau } \wedge m^{-1}$, the operator $\phi$ is a contraction of ${\mathcal {B}}_\tau$ into itself. Thus, $\phi$ admits a unique fixed point in ${\mathcal {B}}_\tau$, which is solution to (A.4) on ${\mathcal {D}}_\tau$. $\square$

Lemma A.2

Let $\xi = (\xi _1, \xi _2, \xi _3)$ denote the absolutely continuous solution of (A.4) on ${\mathcal {D}}_{\tau }$ from Lemma A.1. Then $\xi$ is Lipschitz in each variable on ${\mathcal {D}}_\tau$.

Proof

As $\xi _1$, $\xi _2$ and $\xi _3$ are continuous on ${\mathcal {D}}_\tau$, there exists a constant $m>0$ such that $|\xi _1| \wedge |\xi _2| \wedge |\xi _3| \le m$ on ${\mathcal {D}}_\tau$. Thus, $\xi _1$ is Lipschitz with constant $\kappa =m^2\sigma ^{-2}$. Let us now show that $\xi _2$ and $\xi _3$ are Lipschitz in the s-variable. Fix $t \in [T-d - \tau , T-d]$ and $\eta >0$. Then, for any $s \in [-d, 0]$ such that $s+\eta \in [-d, 0]$, we have

$$\begin{aligned} \left| \xi _2(t,s) - \xi _2(t,s+\eta ) \right|&\le \kappa \eta +{\sigma ^{-2}} \Bigg | \int _{t}^{(T-d) \wedge (t+s+\eta +d)}\\&\quad \times \xi _2(x,0)\xi _3(x,t+s+\eta -x,0)\mathrm{d}x\\&\quad - \int _{t}^{(T-d) \wedge (t+s+d)} \xi _2(x,0)\xi _3(x,t+s-x,0)\mathrm{d}x \Bigg | \\&\le \kappa \eta + \mathbf{I}(t,s) + \mathbf{II}(t,s), \end{aligned}$$

Since $|\xi _2| \le m$, it yields

$$\begin{aligned} \mathbf{I}(t,s)&\le \int _t^{(T-d) \wedge (t+s+d)} \big |\xi _2(x,0)\big | \big | \xi _3(x,t+s+\eta -x,0) - \xi _3(x,t+s-x,0) \big |\mathrm{d}x \\&\le m \int _t^{(T-d) \wedge (t+s+d)} \epsilon (x) \mathrm{d}x, \end{aligned}$$

where $\epsilon$ is defined as

$$\begin{aligned} \epsilon (x) =\underset{\begin{array}{c} s,r\\ \in [-d,0]^2 \end{array}}{\sup } \left| \xi _3(x,s,r)- \xi _3(x,s + \eta ,r) \right| + \underset{s\in [-d,0]}{\sup }\left| \xi _2(x,s)-\xi _2(x,s + \eta ) \right| . \end{aligned}$$

Furthermore, as $|\xi _2| \wedge |\xi _3| \le m$ on ${\mathcal {D}}_\tau$, we have

$$\begin{aligned} \mathbf{II}(t,s)&\le \int _{(T-d) \wedge (t+s+d) }^{(T-d) \wedge (t+s+\eta +d)} |\xi _2(x,0)\xi _3(x,t+s+\eta -x,0)|\mathrm{d}x \\&\le m^2 \eta . \end{aligned}$$

Consequently, for any $t \in [T-d-\tau , T-d]$, we obtain

$$\begin{aligned} \underset{s}{\sup }\left| \xi _2(t,s)- \xi _2(t,s+\eta ) \right| \le m^2 \eta + m \int _t^{T-d} \epsilon (r)\mathrm{d}r. \end{aligned}$$

(A.5)

Looking at the equation of $\xi _3$ in system (A.4), we obtain in a similar manner

$$\begin{aligned} \left| \xi _3(t,s,r)-\xi _3(t,s+\eta ,r) \right| \le&|b| \mathbf{I}(t,s,r) +\sigma ^{-2}{} \mathbf{II}(t,s,r). \end{aligned}$$

(A.6)

An application to the triangle inequality combined with (A.5) and the Lipschitzianity of $\xi _1$ leads to

$$\begin{aligned} \mathbf{I}(t,s,r)&\le |\xi _2((T-d) \wedge (t+(s+\eta ) \wedge r+d), (s+\eta -r) \vee (r-(s+\eta ))-d) \nonumber \\&\quad - \xi _2((T-d) \wedge (t+s \wedge r+d),(s-r) \vee (r-s)-d)|\nonumber \\&\le (\kappa + m^2(1+ \sigma ^{-2})) \eta + m \int _{(T-d) \wedge (t+s \wedge r+d)}^{T-d} \epsilon (x)\mathrm{d}x\nonumber \\&\le (1+2\kappa ) \eta + m \int _{t}^{T-d} \epsilon (x)\mathrm{d}x. \end{aligned}$$

(A.7)

Furthermore

$$\begin{aligned} \mathbf{II}(t,s,r)&\le \Big |\int _{t}^{(T-d) \wedge (t+(s+\eta ) \wedge r+d)} \xi _3(x,t+s+\eta -x,0)\xi _3(x,0,t+r-x)\mathrm{d}x\nonumber \\&\quad -\int _{t}^{(T-d) \wedge (t+s \wedge r+d)} \xi _3(x,t+s-x,0)\xi _3(x,0,t+{ r}-x)\mathrm{d}x \Big |\nonumber \\&\le \int _{t}^{(T-d) \wedge (t+s \wedge r+d)} |\xi _3(x,0,t+r-x)| |\xi _3(x,t+s-x,0) \nonumber \\&\quad - \xi _3(x,t+(s+\eta )-x,0)| \mathrm{d}x\nonumber \\&\quad + \int ^{(T-d) \wedge (t+(s+\eta ) \wedge r+d)}_{(T-d) \wedge (t+s \wedge r+d)} | \xi _3(x,t+(s+\eta )-x,0)\xi _3(r,0,t+r-x) | \mathrm{d}x\nonumber \\&\le m^2 \eta + \int _t^{T-d} \epsilon (r) \mathrm{d}r \end{aligned}$$

(A.8)

Thus, inequality (A.7) together with (A.8) and (A.6) yield the existence of a positive constant $c>0$, independent of $\eta$, such that

$$\begin{aligned} \sup _{\begin{array}{c} s,r \\ \in [-d,0]^2 \end{array}} \left| \xi _3(t,s,r) -\xi _3(t,s+\eta ,r) \right| \le&c \left( \eta + \int _t^{T-d} \epsilon (r) \mathrm{d}r \right) , \end{aligned}$$

which, combined with (A.5) leads, for any $t\in [T-d-\tau , T-d]$, to

$$\begin{aligned} \epsilon (t) \le c \left( \eta + \int _t^{T-d} \epsilon (r)\mathrm{d}r \right) . \end{aligned}$$

Consequently, an application to Gronwall’s lemma yields $\epsilon (t) \le m' \eta$ on $[T-d-\tau , T-d]$, with $m'>0$. Thus, $\xi _2$ and $\xi _3$ are Lipschitz in the s-variable. The arguments for showing that $\xi _2$ and $\xi _3$ are Lipschitz in the t-variable and $\xi _3$ Lipschitz in the r-variable follow the same line. $\square$

Lemma A.3

There exists a unique absolutely continuous solution $\xi =(\xi _1, \xi _2, \xi _3)$ of (A.4) on $[T-2 d, T-d]\times [-d,0]^2$ such that $\xi _1\ge 1- d \left( \frac{b}{\sigma } \right) ^2 >0$.

Proof

Let $\theta \in [T - 2d, T - d)$ denote the lower limit of all $\tau$’s such that there exists an absolutely continuous solution $(\xi _1, \xi _2, \xi _3)$ to (A.4) on $[\theta , T-d]$. Assume $\theta > T - 2 d$. From Lemma A.2, $\xi _1$, $\xi _2$ and $\xi _3$ are Lipschitz in each variable and thus admit a limit, when $t \rightarrow \theta$, which is Lipschitz. Therefore, the argument of Lemma A.1 can be repeated to extend the existence and uniqueness of the solution of system (A.4) on $[\xi , T-d]$ for $T-2d \le \xi < \theta$. As a result, we necessarily have $\theta = T - 2 d$. It remains to prove that $0<\xi _1$. For this, note that since $\xi _1$ is solution to (A.4), we have

$$\begin{aligned} {\Vert }\xi _1 - 1\Vert _\infty \le \frac{d}{\sigma ^{2}} \sup _{\begin{array}{c} t \in \\ {[}T-2 d, T-d{]} \end{array}} |\xi _2(t,0)|^2. \end{aligned}$$

(A.9)

By injecting the boundary condition (A.2) into the system (A.4), one notes that $t\in [T-2 d, T-d] \mapsto \xi _2(t, 0)$ is solution to

$$\begin{aligned} \xi _2(t, 0) = b - \sigma ^{-2}\int _t^{T-d} \xi _2(x, 0)\xi _3(x,t-x, 0) \mathrm{d}x, \qquad T-2 d \le t \le T-d. \end{aligned}$$

But for every $t \in [T-2d , T-d]$, $f_t : x\in [t, T-d] \mapsto f_t(x) := \xi _3(x,t-x,0)$ takes only positive values as $f_t$ is solution to the system

$$\begin{aligned} f_t(x)&= b^2 - \sigma ^{-2}\int _x^{T-d} f_t(u) \xi _3(u, 0,x-u) \mathrm{d}u, \qquad x \in [t,T-d],\\ f_t(T-d)&= b^2, \end{aligned}$$

which can be proven to admit, through a contraction proof in the Banach space $C([t,T-d], {\mathbb {R}})$, a unique positive solution since $\xi$ and its derivatives are bounded. Similarly, we also have $\xi _2(t,0) \ge 0$ for any $t \in [T-2d, T-d]$. As a result, we have $\textit{sign}(\xi _2) = \textit{sign}(b)$ and

$$\begin{aligned} \sup _{\begin{array}{c} t \in \\ {[}T-2 d, T-d{]} \end{array}} {|}\xi _2(t,0)| \le |b|. \end{aligned}$$

(A.10)

Consequently, (A.9) and (A.10) yield that for any $T-2 d \le t \le T-d$, we have $\xi _1 \ge 1- d \left( \frac{b}{\sigma } \right) ^2 = a_2 >0$ as ${\mathcal {N}}(d, b, \sigma )$ is assumed to be greater than 2. $\square$

Finally, by setting $P_{11}(t) = \xi _1(t)$, $P_{12}(t,s) =\xi _2(t,s)$, $P_{22}(t,s,r)=\xi _3(t,s,r)$ and $P_{\hat{22}}(t,s) =\xi _1(t+s+d)$ for any $(t,s,r) \in [T-2 d, T-d] \times [-d, 0]^2$, Lemma A.3 yields the existence and uniqueness of a solution P to (2.9)–(2.10)–(2.11) in the sense of Definition 2.6 on $[T-2 d, T-d]$. The concatenation of the unique solution of (2.9)–(2.10)–(2.11) on $[T-d, T]$ and $[T-2d, T-d]$ leads to a unique solution on $[T-2d, T]$.

1.3 A.3 From Slice $[T- n d, T]$ to $[T-(n+1)d, T]$

Let n be an integer such that $2 \le n < {\mathcal {N}}(d, b, \sigma )$. Assume that there exists a solution P to (2.9)–(2.10)–(2.11) in the sense of Definition 2.6 on $[T-nd, T]$ such that $0<a_n \le P_{11}(t) \le 1$, for any $t \ge T-nd$. Recall the Definition 3.7 of $(a_n)_{n \ge 0}$. Consider the following system on $[T-(n+1)d, T-n d]\times [-d, 0]^2$

$$\begin{aligned} P_{11}(t)&= P_{11}(T-nd)- \int _{t}^{T-nd} \frac{P_{12}(x,0)^{2}}{\sigma ^2 P_{11} (x+d)}\mathrm{d}x,\nonumber \\ P_{12}(t,s)&= b P_{11}((T-nd) \wedge (t+s+d))\nonumber \\&\quad -\int _{t}^{(T-n d) \wedge (t+s+d)} \frac{P_{12}(x,0)P_{22}(x,t+s-x,0)}{\sigma ^2 P_{11} (x+d)}\mathrm{d}x,\nonumber \\ P_{22}(t,s,r)&= b P_{12}((T-nd) \wedge (t+s \wedge r+d),(s-r) \vee (r-s)-d)\nonumber \\&\quad -\int _{t}^{(T-nd) \wedge (t+s \wedge r+d)} \frac{P_{22}(x,t+s-x,0) P_{22}(x,0,t+r-x)}{\sigma ^2 P_{11} (x+d)}\mathrm{d}x. \end{aligned}$$

(A.11)

Note that this system is the same as (A.4), the only difference being the term $x\in [T-(n+1)d, T-nd] \mapsto P_{11}(x+d)$ which comes from the previous slice $[T-nd, T-(n-1)d]$. Therefore, it can be considered as a positive continuous coefficient by induction hypothesis. As result, existence and uniqueness on $[T-(n+1)d, T-nd]$ can be proven in the same fashion as in Lemmas A.1–A.2–A.3. It remains to prove that $P_{11}(t)\ge a_{n+1}$ for any $t\in [T-(n+1)d, T-nd]$. As in Lemma A.3, and by using the induction hypothesis, we have

$$\begin{aligned} |P_{12}(t, -d)| \le |bP_{11}(T-nd)| \le |b|, \qquad t \in [T-(n+1)d, T-nd]. \end{aligned}$$

(A.12)

Furthermore, $P_{11}$ satisfies (A.11) on $[T-(n+1)d, T-nd]$, which, combined with $P_{11} \ge a_n$ on $[T-nd, T-(n-1)d]$ and (A.12) yields

$$\begin{aligned} P_{11}(t)&\ge P_{11}(T-nd) - \frac{d}{a_n} \left( \frac{b}{\sigma }\right) ^2 \\&\ge a_n - \frac{d}{a_n} \left( \frac{b}{\sigma }\right) ^2 = a_{n+1}>0, \end{aligned}$$

for any $t \in [T-(n+1) d, T-n d]$, which ends the proof.