An Efficient Numerical Algorithm for Solving Data Driven Feedback Control Problems

Abstract

The goal of this paper is to solve a class of stochastic optimal control problems numerically, in which the state process is governed by an Itô type stochastic differential equation with control process entering both in the drift and the diffusion, and is observed partially. The optimal control of feedback form is determined based on the available observational data. We call this type of control problems the data driven feedback control. The computational framework that we introduce to solve such type of problems aims to find the best estimate for the optimal control as a conditional expectation given the observational information. To make our method feasible in providing timely feedback to the controlled system from data, we develop an efficient stochastic optimization algorithm to implement our computational framework.

Appendix: Derivation for the Gradient Process

In this “Appendix”, we give a detailed discussion on the derivation for the gradient process (8). To proceed, let U be convex and $(X^*,u^*)$ be any state-control pair which could be an optimal pair. For any $u^M \in \mathcal{U}_{ad}[0,T]$, we let $u^{\varepsilon ,M}=u^*+\varepsilon (u^M-u^*)$. Then the Gâteaux derivative of $u^M\mapsto J^*(u^M)$ at $u^*$ is given by the following:

$$\begin{aligned} \begin{array}{ll} \displaystyle \lim _{\varepsilon \rightarrow 0}{J^*(u^{\varepsilon ,M} )-J^*(u^*)\over \varepsilon }= {\mathbb {E}} \Big [\int _0^T\big (f^{*}_x(t) \mathcal {D}X_t+f^{*}_u(t)[u^M_t-u^{*}_t]\big )d t+h_x\mathcal{D}X_T\Big ], \end{array} \end{aligned}$$

(46)

where

$$\begin{aligned} \begin{array}{ll} \displaystyle \mathcal{D}X_t=\int _0^t\Big (b^{*}_x(s)\mathcal{D}X_s+b^{*}_u(s)[u^M_s-u^*_s]\Big )ds+\int _0^t\Big (\sigma ^{*}_x(s)\mathcal{D}X_s+\sigma ^{*}_u(s)[u^M_s-u^*_s]\Big )dW_s, \end{array} \end{aligned}$$

with $\mathcal{D}X_0 = 0$. For convenience of presentation, we denote $\psi ^{*}(t) := \psi (t,X^*_t,u^*_t)$ and use subscript to denote partial derivative of a function. Let $(Y,Z,\zeta )$ be the adapted solution to the following adjoint backward stochastic differential equation (BSDE)

$$\begin{aligned} \displaystyle dY_s=\Big (-b^{*}_x(s)^\top Y_s-\sigma ^{*}_x(s)^\top Z_s-g^{*}_x(s)^\top \Big )ds+Z_s dW_s+\zeta _sdB_s,\qquad Y_T=(h^{*}_x)^\top \end{aligned}$$

with $h^{*}_x:=h_x(X^*_T)$, where Z is the martingale representation of Y with respect to W and $\zeta$ is the martingale representation of Y with respect to B. Then

$$\begin{aligned} \begin{aligned} \displaystyle h^{*}_x \mathcal {D}X_T=&\ \langle Y_T, \mathcal {D}X_T\rangle -\langle Y_0, \mathcal {D}X_0\rangle \\ \displaystyle =&\ \int _0^T\Big (\langle -b^{*}_x(t)^\top Y_t-\sigma ^{*}_x(t)^\top Z_t-f^{*}_x(t)^\top ,\mathcal {D}X_t\rangle \\&\ \quad \quad \quad \quad +\langle Y_t,b^{*}_x(t)\mathcal {D}X_t+b^{*}_u(t)[u^M_t-u^*_t]\rangle \\&\ \displaystyle \qquad \qquad +\langle Z_t,\sigma ^{*}_x(t)\mathcal {D}X_t+\sigma ^{*}_u(t)[u^M_t-u^*_t]\rangle \Big )ds\\&\ \displaystyle \qquad \qquad +\int _0^T\Big (\langle Z_t, \mathcal {D}X_t\rangle +\langle Y_t, \sigma ^{*}_x(t)\mathcal {D}X_t+\sigma ^{*}_u(t)[u^M_t-u^{*}_t]\Big )dW_t \\&\ \displaystyle \qquad \qquad +\int _0^T\langle \zeta _t,\mathcal{D}X_t+\sigma ^{*}_u(t)[u^M_t-u_t^*]\rangle dB_t\\ \displaystyle =&\ \int _0^T\Big (-f^{*}_x(t)\mathcal {D}X_t+\langle b^{*}_u(t)^\top Y_t+\sigma ^{*}_u(t)^\top Z_t,u^M_t-u^*_t\rangle \Big )dt\\&\ \displaystyle \qquad +\int _0^T\Big (\langle Z_t, \mathcal {D}X_t\rangle +\langle Y_t,\sigma ^{*}_x(t)\mathcal {D}X_t\rangle +\sigma ^{*}_u(t)[u^M_t-u^*_t]\Big )dW_t\\&\ \displaystyle \qquad \qquad +\int _0^T\langle \zeta _t,\mathcal{D}X_t+\sigma ^{*}_u(t)[u^M_t-u_t^*]\rangle dB_t. \end{aligned} \end{aligned}$$

Substituting the above equation into the right hand side of (46), we obtain

$$\begin{aligned} \begin{aligned}&\lim _{\varepsilon \rightarrow 0}{J^*(u^{\varepsilon , M})-J^*(u^*)\over \varepsilon }\\ =&\ {\mathbb {E}} \Big [\int _0^T\big (f_x(t)\mathcal {D}X_t+f_u(t)[u^M_t-u^*_t]\big )dt+h_x\mathcal{D}X_T \Big ]\\ =&\ {\mathbb {E}} \Big [\int _0^T\Big (\langle b^{*}_u(t)^\top Y_t+\sigma ^{*}_u(t)^\top Z_t+f^{*}_u(t)^\top ,u^M_t-u^*_t\rangle \Big )dt\Big ]\\ =&\ {\mathbb {E}} \Big [\int _0^T\langle {\mathbb {E}} \big [b^{*}_u(t)^\top Y_t+\sigma ^{*}_u(t)^\top Z_t+f^{*}_u(t)^\top \bigm |\mathcal{F}^M_t\big ],u^M_t-u^*_t\rangle dt\Big ]. \end{aligned} \end{aligned}$$

Here, we have used that $u^M_t-u_t^*$ is $\mathcal{F}_t^M$-measurable. Hence, in the case that $u^*$ is in the interior of U, one has

$$\begin{aligned} (J^*)'_u(u^*_t)= {\mathbb {E}} \big [b^{*}_u(t)^\top Y_t+\sigma ^{*}_u(t)^\top Z_t+f^{*}_u(t)^\top \bigm |\mathcal{F}^M_t\big ], \quad t \in [0, T], \end{aligned}$$

as required in (8), where Y and Z are solutions of the FBSDE system (9).