Abstract
The classical Method of Successive Approximations (MSA) is an iterative method for solving stochastic control problems and is derived from Pontryagin’s optimality principle. It is known that the MSA may fail to converge. Using careful estimates for the backward stochastic differential equation (BSDE) this paper suggests a modification to the MSA algorithm. This modified MSA is shown to converge for general stochastic control problems with control in both the drift and diffusion coefficients. Under some additional assumptions the rate of convergence is shown. The results are valid without restrictions on the time horizon of the control problem, in contrast to iterative methods based on the theory of forward-backward stochastic differential equations.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Introduction
Stochastic control problems appear naturally in a range of applications in engineering, economics and finance. With the exception of very specific cases such as the linear-quadratic control problem in engineering or Merton portfolio optimization task in finance, stochastic control problems typically have no closed form solutions and have to be solved numerically. In this work, we consider a modification to the method of successive approximations (MSA), see Algorithm 1. The MSA is essentially a way of applying the Pontryagin’s optimality principle to get numerical solutions of stochastic control problems.
We will consider the continuous space, continuous time problem where the controlled system is modelled by an \(\mathbb R^d\)-valued diffusion process. Let W be a \(d'\)-dimensional Wiener martingale on a filtered probability space \((\Omega , \mathcal {F}, (\mathcal {F}_t)_{t\ge 0}, \mathbb {P})\). We will provide exact assumptions we need in Sect. 2. For now, let us fix a finite time \(T\in (0,\infty )\) and consider the controlled stochastic differential equation (SDE) for given measurable functions \(b:[0,T]\times \mathbb {R}^d\times A\rightarrow \mathbb {R}^d\) and \(\sigma :[0,T]\times \mathbb {R}^d\times A\rightarrow \mathbb {R}^{d\times d'}\)
Here \(\alpha =(\alpha _s)_{s\in [0,T]}\) is a control process belonging to the space of admissible controls \(\mathcal A\), valued in a separable metric space A and we will write \(X^{\alpha }\) to denote the unique solution of (1) which starts from x at time 0 whilst being controlled by \(\alpha \). Furthermore let \(f:[0,T]\times \mathbb R^d \times A \rightarrow \mathbb R\) and \(g:\mathbb R^d \rightarrow \mathbb R\) be given measurable functions and consider the gain functional
for all \(x\in \mathbb {R}^d\) and \(\alpha \in \mathcal {A}\). We want to solve the optimisation problem i.e. to find the optimal control \(\alpha ^*\) which achieves the minimum of (2) (or, if the infimum cannot be reached by \(\alpha \in \mathcal A\) then an \(\varepsilon \)-optimal control \(\alpha ^\varepsilon \in \mathcal A\) such that \(\inf _{\alpha \in \mathcal {A}}J(x,\alpha ) \le J(x,\alpha ^\varepsilon ) + \varepsilon \)).
In the present paper, we study an approach based on Pontryagin’s optimality principle, see e.g. [4, 7] or [25]. The main idea is to consider optimality conditions for controls of the problem (2). Given \(b, \sigma \) and f we define the Hamiltonian \(\mathcal {H}:[0,T]\times \mathbb {R}^d\times \mathbb {R}^d\times \mathbb {R}^{d\times d'}\times A\rightarrow \mathbb {R}\) as
Consider for each \(\alpha \in \mathcal {A}\), the BSDE, called the adjoint equation
It is well known from Pontryagin’s optimality principle that, if an admissible control \(\alpha ^*\in \mathcal {A}\) is optimal, \(X^{\alpha ^*}\) is the corresponding optimally controlled dynamic (1) and \((Y^{\alpha ^*},Z^{\alpha ^*})\) is the solution to the associated adjoint equation (4), then \(\forall a\in A\) and \(\forall s\in [0,T]\) the following holds
We now define the augmented Hamiltonian \(\mathcal {\tilde{H}}:[0,T]\times \mathbb {R}^d\times \mathbb {R}^d\times \mathbb {R}^{d\times d'}\times A\times A\rightarrow \mathbb {R}\) for some \(\rho \ge 0\) by
Notice that when \(\rho =0\) we have exactly the definition of Hamiltonian (3). Given the augmented Hamiltonian, let us introduce the modified MSA in Algorithm 1 which consists of successive integrations of the state and adjoint systems and updates to the control. Notice that the backward SDE depends on the Hamiltonian \(\mathcal {H}\), while the control update step comes from minimizing the augmented Hamiltonian \(\tilde{\mathcal {H}}\).
The method of successive approximations (i.e. case \(\rho =0\)) for numerical solution of deterministic control problems was proposed already in [5]. Recent application of the modified MSA to a deep learning problem has been studied in [32], where they formulated the training of deep neural networks as an optimal control problem and introduced the modified method of successive approximations as an alternative training algorithm for deep learning. For us, the main motivation to explore the modified MSA for stochastic control problems is to obtain convergence, ideally with rate, of an iterative algorithm, applicable to problems with the control in the diffusion part of the controlled dynamics. This is in contrast to [36] where convergence rate of an the Bellman–Howard policy iteration is shown but only for control problems with no control in the diffusion part of the controlled dynamics.
In Lemma 2.3, which can be established using careful BSDE estimates, we can see the estimate on the change of J when we do a minimization step of Hamiltonian as in (8). If the sum of the last three terms of (14) is bigger than the first term, then for classical MSA algorithm (i.e. case \(\rho =0\)) we cannot guarantee that we do an update of the control in optimal descent direction of J. That means that the method of successive approximations may diverge. To overcome this, we need to modify the algorithm in such way so that we ensure convergence. With this in mind the desirability of the the augmented Hamiltonian (6) for updating the control becomes clear, as long as it still characterises optimal controls like \(\mathcal H\) does. Theorem 2.4 answers this question affirmatively which opens the way to the modified MSA. In Theorem 2.5 we show that the modified method of successive approximations, converges for arbitrary T, and in Corollary 2.6, we show logarithmic convergence rate for certain stochastic control problems.
We observe that the forward and backward dynamics in (7) are decoupled, due to the iteration used. Therefore, it can be efficiently approximated, even in high dimension, using deep learning methods, see [30, 31]. However, the minimization step (8) might be computationally expensive for some problems. A possible approach circumventing this is to replace the full minimization of (8) by gradient descent. A continuous version of this gradient flow is analyzed in [37].
The main contributions of this paper are the probabilistic proof of convergence of the modified method of successive approximations and establishing convergence rate for a specific type of optimal control problems.
This paper is organised as follow: in Sect. 1.1 we compare our results with existing work. In Sect. 2 we state the assumptions and main results. In Sect. 3 we collect all proofs. Finally, in Appendix 1 we recall an auxiliary lemma which is needed in the proof of Corollary 2.6.
1.1 Related Work
One can solve the stochastic optimal control problem using dynamic programming principle. It is well known, see e.g. Krylov [8], that under reasonable assumptions the value function, defined as infimum of (2) over all admissible controls, satisfies the Bellman partial differential equation (PDE). There are several approaches to solve this nonlinear problem. One may apply a finite difference method to discretise the Bellman PDE and get a high dimensional nonlinear system of equations, see e.g [19] or [22]. Or one may linearize the Bellman PDE and then iterate. The classical approach is the Bellman–Howard policy improvement/iteration algorithm, see e.g. [1, 2] or [3]. The algorithm is initialised with a “guess” of Markovian control. Given a Markovian control strategy at step n one solves a linear PDE with the given control fixed and then one uses the solution to the linear PDE to update the Markovian control, see e.g. [27, 28] or [29]. In [36], a global rate of convergence and stability for the policy iteration algorithm has been established using backward stochastic differential equations (BSDE) theory. However, the result only applies to stochastic control problems with no control in the diffusion coefficient of the controlled dynamics.
It is known that the solution of the stochastic optimal control problem can be obtained from a corresponding forward backward stochastic differential equation (FBSDE) via the stochastic optimality principle, see [26, Chap. 8.1]. Indeed, let us consider (1) and (4), and recall from the stochastic optimality principle, see [25, Theorem 4.12], that for the optimal control \(\alpha ^*=(\alpha _s^*)_{s\in [0,T]}\) we have that (5) holds. Assume that under some conditions on \(b,\sigma \) and f we have that the first order condition stated above uniquely determines \(\alpha ^*\) for \(s\in [0,T]\) by
for some function \(\varphi \). Therefore, after plugging (9) into (1) and (4), we obtain the following coupled FBSDE:
where \((\bar{b},\bar{\sigma })(s,X_s^{\alpha ^*},Y_s^{\alpha ^*},Z_s^{\alpha ^*})=(b,\sigma )(s,X_s^{\alpha ^*},\varphi (s,X_s^{\alpha ^*},Y_s^{\alpha ^*},Z_s^{\alpha ^*}))\) and \(\bar{\mathcal {H}}(s,X_s^{\alpha ^*},Y_s^{\alpha ^*},Z_s^{\alpha ^*})=\mathcal {H}(s,X_s^{\alpha ^*},Y_s^{\alpha ^*},Z_s^{\alpha ^*},\varphi (s,X_s^{\alpha ^*},Y_s^{\alpha ^*},Z_s^{\alpha ^*}))\). It is worth mentioning that when \(\sigma \) does not depend on the control \(\bar{\sigma }\) will depend on forward process and time only. This means that \(\bar{\sigma }\) does not have Y and Z components.
The theory of FBSDE has been studied widely and there are several methods to show the existence and uniqueness result, and a number of numerical algorithms have been proposed based on those methods. First is the method of contraction mapping. It was first studied by Antonelli [9] and later by Pardoux and Tang [15]. The main idea there is to show that a certain map is a contraction, and then to apply a fixed point argument. However, it turns out that this method works only for small enough time horizon T. In the case when \(\bar{\sigma }\) does not depend on Y and Z, having small T is sufficient to get contraction. Otherwise, one needs to assume additionally that the Lipschitz constants of \(\bar{\sigma }\) in z and that of g in x satisfy a certain inequality, see [26, Theorem 8.2.1]. Using the method of contraction mapping one can then implement a Picard-iteration-type numerical algorithm and show exponential convergence for small T. The second method is the Four Step Scheme. It was introduced by Ma et al., see [10], and was later studied by Delarue [17]. The idea is to use a decoupling function and then study an associated quasi-linear PDE. We note that in [10, 17] the forward diffusion coefficient \(\bar{\sigma }\) does not depend on Z. This corresponds to stochastic control problems with the uncontrolled diffusion coefficient. The numerical algorithms based on this method exploits the numerical solution of the associated quasi-linear PDE and therefore faces some limitations for high dimensional problems, see Douglas et al. [12], Milstein and Tretyakov [20], Ma et al. [21] and Delarue and Menozzi [18]. Guo et al. [24] proposed a numerical scheme for high-dimensional quasi-linear PDE associated with the coupled FBSDE when \(\bar{\sigma }\) does not depend on Z, which is based on a monotone scheme and on probabilistic approach. Finally, there is the method of continuation. This method was developed by Hu and Peng [11], Peng and Wu [16] and by Yong [14]. It allows them to show the existence and uniqueness result for arbitrary T under monotonicity conditions on the coefficients, which one would not expect to apply to FBSDEs arising from a control problem as described by (9), (10). Recently, deep learning methods have been applied to solving FBSDEs. In [35], three algorithms for solving fully coupled FBSDEs which have good accuracy and performance for high-dimensional problems are provided. One of the algorithms is based on the Picard iteration and it converges, but only for small enough T. In [34], an alternative algorithm for solving high-dimensional fully coupled FBSDEs based on deep learning was proposed, and the convergence result was shown assuming small T and other structural conditions (sometimes referred to as weak coupling and monotonicity conditions).
2 Main Results
We fix a finite horizon \(T\in (0,\infty )\). Let A be a separable metric space. This is the space where the control processes \(\alpha \) take values. We fix a filtered probability space \((\Omega , \mathcal {F}, \mathbb {F}=(\mathcal {F}_t)_{0\le t\le T}, \mathbb {P})\). Let \(W=(W_t)_{t\in [0,T]}\) be a \(d'\)-dimensional Wiener martingale on this space. By \(\mathbb {E}_t\) we denote the conditional expectation with respect to \(\mathcal {F}_t\). Let \(|\cdot |\) denote any norm in a finite dimensional Euclidean space. By \(\Vert \cdot \Vert _{L^\infty }\) we denote the norm in \(L^{\infty }(\Omega )\). Let \(\Vert Z\Vert _{\mathbb {H}^{\infty }}:=\text {ess}\sup _{(t,\omega )}|Z_t(\omega )|\) for any predictable process Z. We understand the following as \(D_x\sigma =D_{x_l}\sigma ^{ij}\), \(D_x^2b=D^2_{x_lx_n}b^i\) and \(D_x^2\sigma =D^2_{x_lx_n}\sigma ^{ij}\), where \(i,l,n=1,2,\dots ,d\) and \(j=1,2,\dots ,d'\). By \(Z^\top \) we denote the transpose of Z.
The state of the system is governed by the controlled SDE (1). The corresponding adjoint equation satisfies (4).
Assumption 2.1
The functions b and \(\sigma \) are jointly continuous in t and twice differentiable in x. There exists \(K\ge 0\) such that \(\forall x\in \mathbb {R}^d,\forall a\in A,\forall t\in [0,T]\),
Moreover, assume that \(D_x^2\sigma (t,x,a)=0\) \(\forall x\in \mathbb {R}^d,\forall a\in A,\forall t\in [0,T]\).
Clearly the assumption (11) implies that \(\forall x,x'\in \mathbb {R}^d,\forall a\in A,\forall t\in [0,T]\) we have
The assumption that \(D_x^2\sigma (t,x,a)=0\) \(\forall x\in \mathbb {R}^d,\forall a\in A,\forall t\in [0,T]\) is needed so that (21), in the proof of Lemma 3.1, holds. Without this assumption (21) would only hold if we could show that \(\Vert Z^\alpha \Vert _{\mathbb H^\infty } < \infty \). Without additional regularization of the control problem this is impossible. Indeed, with [13, Proposition 5.3] we see that \(Z^\alpha _t\) is a version of \(D_t Y_t^\alpha \) (the Malliavin derivative of \(Y_t^\alpha \)) and \(D_t Y_t^\alpha \) itself satisfies a linear BSDE. However, to obtain the estimates using this representation, one term that arises is \(D_t \alpha _s\) where \(t\in [0,T]\) and \(s\in [t,T]\). So we would need \(\text {ess}\sup _{\omega \in \Omega , t\in (0,T),s\in (t,T)}|D_t \alpha _s(\omega )|<\infty \). This is not necessarily the case here.
Assumption 2.2
The functions f is joinly continuous in t, and f and \(\sigma \) are twice differentiable in x. There is a constant \(K\ge 0\) such that \(\forall x,\forall a\in A,\forall t\in [0,T]\)
Under these assumptions, we can obtain the following estimate.
Lemma 2.3
Let Assumption 2.1 and 2.2 hold. Then for any admissible controls \(\varphi \) and \(\theta \) there exists a constant \(C>0\) such that
The proof will be given in Sect. 3. We now state a necessary condition for optimality for the augmented Hamiltonian.
Theorem 2.4
[Extended Pontryagin’s optimality principle] Let \(\alpha ^*\) be the (locally) optimal control, \(X^{\alpha ^*}\) be the associated controlled state solving (1), and \((Y^{\alpha ^*},Z^{\alpha ^*})\) be the associated adjoint processes solving (4). Then for any \(a\in A\) we have
The proof of Theorem 2.4 will come in Sect. 3. We are now ready to present the main result of the paper.
Theorem 2.5
Let Assumptions 2.1 and 2.2 hold. Then Algorithm 1 converges to a local minimum of (2) for sufficiently large \(\rho >0\).
Theorem 2.5 will be proved in Sect. 3. It can be seen from the proof that \(\rho \) needs to be two times larger than the constant appearing in Lemma 2.3, which itself depends increases with T, d and constants from Assumption 2.1 and 2.2.
We cannot guarantee that the Algorithm 1 converges to the optimal control which minimizes (2), since the extended Pontryagin’s optimality principle, see Theorem 2.4, is the necessary condition for optimality. The sufficient condition for optimality tells us that to get the optimal control we need to assume convexity of the Hamiltonian in state and control variables, and need to assume convexity of the terminal cost function. To that end, we need to assume convexity of \(b,\sigma , f\) and g in x and a.
In the following corollary, we show that under a particular setting of the problem we have logarithmic convergence of the modified method of successive approximations to the true solution of the problem.
Corollary 2.6
Let Assumptions 2.1 and 2.2 hold. Moreover, if \(b,\sigma \) and f are in the form of
for \(\forall t\in [0,T],\;\forall x\in \mathbb {R}^d,\;\forall a\in A\). In addition, assume that f and g are convex in x, \(f_2, b_2,\sigma _2\) are convex in a. Then we have the following estimate for the sequence \((\alpha ^n)_{n\in \mathbb {N}}\) from Algorithm 1:
where \(\alpha ^*\) is the optimal control for (2) and C is a positive constant.
The proof of Corollary 2.6 will be given in Sect. 3. Theorem 2.5 and Corollary 2.6 are extensions of the result in [5] to the stochastic case.
3 Proofs
We start working towards the proof of Theorem 2.5. Recall the adjoint equation for an admissible control \(\alpha \):
From now on, we shall use Einstein notation, so that repeated indices in a single term imply summation over all the values of that index.
Lemma 3.1
Assume that there exists \(K\ge 0\) such that \(\forall x\in \mathbb {R}^d,\forall a\in A,\forall t\in [0,T]\) we have
and
Then \(\Vert Y^{\alpha }\Vert _{\mathbb {H}^{\infty }}\) is bounded.
Proof
From the definition of the Hamiltonian (3) we have
Hence, one can observe that (16) is a linear BSDE. Therefore, from [33, Proposition 3.2] we can write the formula for the solution of (16):
where the process S is the unique strong solution of
and \(S^{-1}\) is the inverse process of S. Thus, due to [33, Corollary 3.7] and assumptions of lemma we have the following bound:
Hence, due to assumptions of lemma we conclude that \(\Vert Y^{\alpha }\Vert _{\mathbb {H}^{\infty }}\) is bounded.
Proof of Lemma 2.3 Let \(\varphi \) and \(\theta \) be some generic admissible controls. We will write \((X^{\varphi }_s)_{s\in [0,T]}\) for the solution of (1) controlled by \(\varphi \) and \((X^{\theta }_s)_{s\in [0,T]}\) for the solution of (1) controlled by \(\theta \). We denote solutions of corresponding adjoint equations by \((Y^{\varphi }_s,\,Z^{\varphi }_s)_{s\in [0,T]}\) and \((Y^{\theta }_s,\,Z^{\theta }_s)_{s\in [0,T]}\). Due to Taylor’s theorem, we note that for some \(R^1(\omega )\in [0,1]\), we have \(\forall \omega \in \Omega \) that
The last inequality holds due to Assumption 2.2. Recall that \(Y^\theta _T=D_xg(X^\theta _T)\). Hence, using Itô’s product rule, we get
From this, the forward SDE (1) and the adjoint equation (4) we thus get
On the other hand, by definition of the Hamiltonian we have
Summing up (17) and (18) we get
Due to Taylor’s theorem, there exists \((R^2_s(\omega ))_{s\in [0,T]}\in [0,1]\) such that \(\forall \omega \in \Omega \) we have
Since \(D_x^2\sigma (s,X^\theta _s+R^2_s(X^\varphi _s-X^\theta _s),\varphi _s)=0\) by Assumption 2.1, we have that
From Lemma 3.1 we know that \(|Y^\theta _s|\) is bounded a.s. for all \(s\in [0,T]\). Hence by Assumption 2.1 and 2.2 we have
Therefore, after substituting (20) into (19), and by 21 we get
Let us now get a standard SDE estimate for the difference of \(X^{\varphi }\) and \(X^{\theta }\). From \((a+b)^2\le 2a^2+2b^2\), from taking the expectation, from Hölder’s inequality, from Assumption 2.1, from the Burkholder–Davis–Gundy inequality and from Gronwall’s inequality we obtain
Young’s inequality allows us to get the estimate
Hence, from (22) we have that
for some constant \(C>0\), which depends on K, T, and d. \(\square \)
Proof of Theorem 2.4 Since \(\alpha ^*\) is the (locally) optimal control for the problem (2), the Pontryagin’s optimality principle holds, see e.g. [23]. Hence for any \(a\in A\) we have
By definition of the augmented Hamiltonian (6) for all \(s\in [0,T]\) we have
Therefore, due to (23) and (24) we have
This concludes the proof.\(\square \)
Proof of Theorem 2.5 Let us apply Lemma 2.3 for \(\varphi =\alpha ^{n}\) and \(\theta =\alpha ^{n-1}\). Hence, for some \(C>0\) we have
Let
Due to the definition of \(\alpha ^n\) (8) and (15) we have for all \(s\in [0,T]\)
Therefore, we can observe that \(\mu (\alpha ^{n-1})\le 0\). Hence we can rewrite the inequality (25) as
where \(D:=1-\frac{2C}{\rho }\). By choosing \(\rho >2C\) we have that \(D>0\). Notice that for any integer \(M>1\) we have
Since \((-\mu (\alpha ^{n-1}))\ge 0\) and \(\sum _{n=1}^\infty (-\mu (\alpha ^{n-1}))<+\infty \) we have that \(\mu (\alpha ^{n-1})\rightarrow 0\) as \(n\rightarrow 0\). This concludes the proof. \(\square \)
We need to introduce new notation, which will be used in the proof of Corollary 2.6. Denote the set
Let us define for all \(s\in [0,T]\)
and
By definition of \(\alpha ^n\) notice that \(\Delta _{\alpha ^{n-1}} \mathcal {H}(t)\le 0\) for all \(t\in [0,T]\). Let us show an auxiliary lemma.
Lemma 3.2
For any \(h>0\) there exists \(\tau \), which depends on h and \(\alpha ^{n-1}\), such that
Proof
We will prove by contradiction. Assume that there exists \(h^*>0\) such that \(\forall \tau \in [0,T]\) we have
Denote \(\tau _i=ih^*\), \(i=1,\dots ,N(h^*)\), where \(N(h^*)=[T/h^*]\) - integer part. Since \(\Delta _{\alpha ^{n-1}} \mathcal {H}(t)\le 0\) for all \(t\in [0,T]\) by definition of \(\alpha ^n\) and \(\cup _{i=1}^{N(h^*)}I_{\tau _i, h^*}\) is a superset of [0, T] we have
Hence, by (28) we get
Last inequality holds since \(\frac{h^* N(h^*)}{T}\) is less or equal to 1. Hence we get the contradiction. \(\square \)
Now we are ready to prove Corollary 2.6.
Proof of Corollary 2.6 First, observe that
Let us consider the set \(I_{\tau , h}\) given by (27). We will specify the choice of \(\tau \) and h later. Hence, after applying Lemma 2.3 for \(\alpha ^n\) and \(\alpha ^{n-1}\) we have for some \(C>0\)
Since the following holds for all \(s\in [0,T]\) and \(\rho \ge 0\):
we have for \(\rho \ge 2C\)
Therefore, from Lemma 3.2 and from similar calculations as in (26), there exists \(\tau \) such that
Let us choose \(h=-(\rho -2C)\mu (\alpha ^{n-1})/(\rho T)\). Hence
Let \(\alpha ^*\) be the optimal control. Indeed, by the sufficient condition for optimality, see e.g. [23], and by assumptions of corollary, we have the existence of the optimal control. Therefore, by convexity of g, and by Itô’s product rule we have
Hence, we have that
Recalling the form of \(b,\sigma \) and observing that
we have
Since f is convex in x we have for all \(s\in [0,T]\) that
Therefore, we obtain
where the second inequality holds due to
Let \(b^n:=J(x,\alpha ^n)-J(x,\alpha )\), then due to (30) and (31) we have that
Therefore, due to Lemma A.1 we have
for some constant \(C_1>0\). This concludes the proof.\(\square \)
References
Antonelli, F.: Backward-forward stochastic differential equations. Ann. Appl. Probab. 3, 777–793 (1993)
Bellman, R.: Functional equations in the theory of dynamic programming. V. Positivity and quasi-linearity. Proc. Natl Acad. Sci. U.S.A. 41(10), 743–746 (1955)
Bellman, R.: Dynamic Programming. Princeton University Press, Princeton (1957)
Boltyanskii, V.G., Gamkrelidze, R.V., Pontryagin, L.S.: Theory of optimal processes. I. The maximum principle. Izv. Akad. Nauk SSSR Ser. Mat. 24(1), 3–42 (1960)
Carmona, R.: Lectures on BSDEs, Stochastic Control, and Stochastic Differential Games with Financial Applications. SIAM, Philadelphia (2016)
Delarue, F.: On the existence and uniqueness of solutions to FBSDEs in a non-degenerate case. Stoch. Process. Appl. 99, 209–289 (2002)
Delarue, F., Menozzi, S.: A forward-backward stochastic algorithm for quasi-linear PDEs. Ann. Appl. Probab. 16(1), 140–184 (2006)
Demyanov, V.D., Rubinov, A.M.: Approximate Methods in Extremal Problems. Leningrad State University, Leningrad (1968). (in Russian)
Dong, H., Krylov, N.V.: The rate of convergence of finite-difference approximations for parabolic Bellman equations with Lipschitz coefficients in cylindrical domains. Appl. Math. Optim. 56(1), 37–66 (2007)
Douglas, J., Ma, J., Protter, P.: Numerical methods for forward-backward stochastic differential equations. Ann. Appl. Probab. 6(3), 940–968 (1996)
El Karoui, N., Peng, S., Quenez, M.C.: Backward stochastic differential equations in finance. Math. Finance 7(1), 1–71 (1997)
Guo, W., Zhang, J., Zhuo, J.: A monotone scheme for high-dimensional fully nonlinear PDEs. Ann. Appl. Probab. 25(3), 1540–1580 (2015)
Gyöngy, I., Šiska, D.: On finite-difference approximations for normalized Bellman equations. Appl. Math. Optim. 60, 297–339 (2009)
Han, J., Long, J.: Convergence of the deep BSDE method for coupled FBSDEs. Probab. Uncertain. Quant. Risk. 5(1), 1–33 (2020)
Han, J., Jentzen, A., Weinan, E.: Solving high-dimensional partial differential equations using deep learning. Proc. Natl. Acad. Sci. U.S.A. 115(34), 8505–8510 (2018)
Harter, J., Richou, A.: A stability approach for solving multidimensional quadratic BSDEs. Electron. J. Probab. 24(4), 1–51 (2019)
Howard, R.A.: Dynamic Programming and Markov Processes. MIT Press, Cambridge (1960)
Hu, Y., Peng, S.: Solution of forward-backward stochastic differential equations. Probab. Theory Relat. Fields 103, 273–283 (1995)
Jacka, S.D., Mijatović, A.: On the policy improvement algorithm in continuous time. Stochastics 89(1), 348–359 (2017)
Jacka, S.D., Mijatović, A., Siraj, A.: Coupling and a generalised Policy Iteration Algorithm in continuous time. arXiv:1707.07834 (2017)
Ji, S., Peng, S., Peng, Y., Zhang, X.: Three algorithms for solving high-dimensional fully-coupled FBSDEs through deep learning. IEEE Intell. Syst. 35(3), 71–84 (2020)
Kerimkulov, B., Šiška, D., Szpruch, Ł.: Exponential convergence and stability of Howard’s policy improvement algorithm for controlled diffusions. SIAM J. Control Optim. 58(3), 1314–1340 (2020)
Krylov, N.V.: Controlled Diffusion Processes. Springer, Berlin (1980)
Krylov, I.A., Chernousko, F.L.: On the method of successive approximations for solution of optimal control problems (in Russian). U.S.S.R. Comput. Math. Math. Phys. 2(6), 1371–1382 (1963)
Li, Q., Chen, L., Tai, C., Weinan, E.: Maximum principle based algorithms for deep learning. J. Mach. Learn. Res. 18(165), 1–29 (2018)
Ma, J., Protter, P., Yong, J.M.: Solving forward-backward stochastic differential equations explicitly—a four step scheme. Probab. Theory Relat. Fields 98, 339–359 (1994)
Ma, J., Shen, J., Zhao, Y.: On numerical approximations of forward-backward stochastic differential equations. SIAM J. Numer. Anal. 46(5), 2636–2661 (2008)
Maeda, A., Jacka, S.D.: Evaluation of the rate of convergence in the PIA. arXiv:1709.06466 (2017)
Milstein, G., Tretyakov, M.: Discretization of forward-backward stochastic differential equations and related quasi-linear parabolic equations. IMA J. Numer. Anal. 27(1), 24–44 (2007)
Pardoux, E., Tang, S.: Forward-backward stochastic differential equations and quasilinear parabolic PDEs. Probab. Theory Relat. Fields 114(2), 123–150 (1999)
Peng, S., Wu, Z.: Fully coupled forward-backward stochastic differential equations and applications to optimal control. SIAM J. Control Optim. 37, 825–843 (1999)
Pham, H.: Continuous-Time Stochastic Conrol and Optimization with Financial Applications. Springer, Berlin (2009)
Pontryagin, L.S.: Mathematical Theory of Optimal Processes. CRC Press, Boca Raton (1987)
Sabate Vidales, M., Šiška, D., Szpruch, Ł.: Unbiased deep solvers for parametric PDEs. arXiv:1810.05094v2 (2018)
Šiška, D., Szpruch, Ł.: Gradient flows for regularized stochastic control problems. arXiv:2006.05956 (2020)
Yong, J.: Finding adapted solutions of forward-backward stochastic differential equations: method of continuation. Probab. Theory Relat. Fields 107, 537–572 (1997)
Zhang, J.: Backward Stochastic Differential Equations: From Linear to Fully Nonlinear Theory. Springer, New York (2017)
Acknowledgements
The authors would like to thank Yufei Zhang (University of Oxford), Adrien Richou (Université de Bordeaux) and the anonymous referees. Their corrections and suggestions helped to improve the paper.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This study was supported by the Alan Turing Institute under EPSRC grant no. EP/N510129/1 and by The Maxwell Institute Graduate School in Analysis and its Applications, a Centre for Doctoral Training funded by the UK Engineering and Physical Sciences Research Council (grant EP/L016508/01), the Scottish Funding Council, Heriot-Watt University and the University of Edinburgh.
Appendix A: Auxiliary Lemma
Appendix A: Auxiliary Lemma
Lemma A.1
Let \(\{b_k\}_{k\in \mathbb {N}}\) be the sequence of nonnegative numbers such that
where q is a positive constant. Then \(b_k=O(1/k)\).
One can find the proof in [6, Lemma 1.4, p. 93]. However, the proof is written in Russian. For convenience of the reader we provide it here.
Proof
Let \(b_k=\frac{c_k}{k}\) for some nonnegative sequence \((c_k)_{k\in \mathbb {N}}\). Then it is enough to show that \(c_k\) is bounded for all \(k\in \mathbb {N}\). By assumption we have
Therefore,
After some transformation, we can rewrite the equation above as
Thus
If \(1-qc_k<0\) we have
Hence \(c_{k+1}<c_k\). On the other hand, if \(1-qc_k\ge 0\), we have \(c_k\le \frac{1}{q}\). Therefore, we conclude that for all k we have
\(\square \)
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Kerimkulov, B., Šiška, D. & Szpruch, L. A Modified MSA for Stochastic Control Problems. Appl Math Optim 84, 3417–3436 (2021). https://doi.org/10.1007/s00245-021-09750-2
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00245-021-09750-2