A Modified MSA for Stochastic Control Problems

The classical Method of Successive Approximations (MSA) is an iterative method for solving stochastic control problems and is derived from Pontryagin’s optimality principle. It is known that the MSA may fail to converge. Using careful estimates for the backward stochastic differential equation (BSDE) this paper suggests a modification to the MSA algorithm. This modified MSA is shown to converge for general stochastic control problems with control in both the drift and diffusion coefficients. Under some additional assumptions the rate of convergence is shown. The results are valid without restrictions on the time horizon of the control problem, in contrast to iterative methods based on the theory of forward-backward stochastic differential equations.


Introduction
Stochastic control problems appear naturally in a range of applications in engineering, economics and finance. With the exception of very specific cases such as the linearquadratic control problem in engineering or Merton portfolio optimization task in finance, stochastic control problems typically have no closed form solutions and have to be solved numerically. In this work, we consider a modification to the method of successive approximations (MSA), see Algorithm 1. The MSA is essentially a way of applying the Pontryagin's optimality principle to get numerical solutions of stochastic control problems.
We will consider the continuous space, continuous time problem where the controlled system is modelled by an R d -valued diffusion process. Let W be a ddimensional Wiener martingale on a filtered probability space ( , F, (F t ) t≥0 , P). We will provide exact assumptions we need in Sect. 2. For now, let us fix a finite time T ∈ (0, ∞) and consider the controlled stochastic differential equation (SDE) for given Here α = (α s ) s∈[0,T ] is a control process belonging to the space of admissible controls A, valued in a separable metric space A and we will write X α to denote the unique solution of (1) which starts from x at time 0 whilst being controlled by α. Furthermore let f : [0, T ] × R d × A → R and g : R d → R be given measurable functions and consider the gain functional for all x ∈ R d and α ∈ A. We want to solve the optimisation problem i.e. to find the optimal control α * which achieves the minimum of (2) (or, if the infimum cannot be reached by α ∈ A then an ε-optimal control α ε ∈ A such that inf α∈A J (x, α) ≤ J (x, α ε ) + ε).
In the present paper, we study an approach based on Pontryagin's optimality principle, see e.g. [4,7] or [25]. The main idea is to consider optimality conditions for controls of the problem (2). Given b, σ and f we define the Hamiltonian Consider for each α ∈ A, the BSDE, called the adjoint equation It is well known from Pontryagin's optimality principle that, if an admissible control α * ∈ A is optimal, X α * is the corresponding optimally controlled dynamic (1) and (Y α * , Z α * ) is the solution to the associated adjoint equation (4), then ∀a ∈ A and ∀s ∈ [0, T ] the following holds We now define the augmented HamiltonianH :  The method of successive approximations (i.e. case ρ = 0) for numerical solution of deterministic control problems was proposed already in [5]. Recent application of the modified MSA to a deep learning problem has been studied in [32], where they formulated the training of deep neural networks as an optimal control problem and introduced the modified method of successive approximations as an alternative training algorithm for deep learning. For us, the main motivation to explore the modified MSA for stochastic control problems is to obtain convergence, ideally with rate, of an iterative algorithm, applicable to problems with the control in the diffusion part of the controlled dynamics. This is in contrast to [36] where convergence rate of an the Bellman-Howard policy iteration is shown but only for control problems with no control in the diffusion part of the controlled dynamics.
In Lemma 2.3, which can be established using careful BSDE estimates, we can see the estimate on the change of J when we do a minimization step of Hamiltonian as in (8). If the sum of the last three terms of (14) is bigger than the first term, then for classical MSA algorithm (i.e. case ρ = 0) we cannot guarantee that we do an update of the control in optimal descent direction of J . That means that the method of successive approximations may diverge. To overcome this, we need to modify the algorithm in such way so that we ensure convergence. With this in mind the desirability of the the augmented Hamiltonian (6) for updating the control becomes clear, as long as it still characterises optimal controls like H does. Theorem 2.4 answers this question affirmatively which opens the way to the modified MSA. In Theorem 2.5 we show that the modified method of successive approximations, converges for arbitrary T , and in Corollary 2.6, we show logarithmic convergence rate for certain stochastic control problems.
We observe that the forward and backward dynamics in (7) are decoupled, due to the iteration used. Therefore, it can be efficiently approximated, even in high dimension, using deep learning methods, see [30,31]. However, the minimization step (8) might be computationally expensive for some problems. A possible approach circumventing this is to replace the full minimization of (8) by gradient descent. A continuous version of this gradient flow is analyzed in [37].
The main contributions of this paper are the probabilistic proof of convergence of the modified method of successive approximations and establishing convergence rate for a specific type of optimal control problems.
This paper is organised as follow: in Sect. 1.1 we compare our results with existing work. In Sect. 2 we state the assumptions and main results. In Sect. 3 we collect all proofs. Finally, in Appendix 1 we recall an auxiliary lemma which is needed in the proof of Corollary 2.6.

Related Work
One can solve the stochastic optimal control problem using dynamic programming principle. It is well known, see e.g. Krylov [8], that under reasonable assumptions the value function, defined as infimum of (2) over all admissible controls, satisfies the Bellman partial differential equation (PDE). There are several approaches to solve this nonlinear problem. One may apply a finite difference method to discretise the Bellman PDE and get a high dimensional nonlinear system of equations, see e.g [19] or [22]. Or one may linearize the Bellman PDE and then iterate. The classical approach is the Bellman-Howard policy improvement/iteration algorithm, see e.g. [1,2] or [3]. The algorithm is initialised with a "guess" of Markovian control. Given a Markovian control strategy at step n one solves a linear PDE with the given control fixed and then one uses the solution to the linear PDE to update the Markovian control, see e.g. [27,28] or [29]. In [36], a global rate of convergence and stability for the policy iteration algorithm has been established using backward stochastic differential equations (BSDE) theory.
However, the result only applies to stochastic control problems with no control in the diffusion coefficient of the controlled dynamics.
It is known that the solution of the stochastic optimal control problem can be obtained from a corresponding forward backward stochastic differential equation (FBSDE) via the stochastic optimality principle, see [26,Chap. 8.1]. Indeed, let us consider (1) and (4), and recall from the stochastic optimality principle, see [25,Theorem 4.12], that for the optimal control α * = (α * s ) s∈[0,T ] we have that (5) holds. Assume that under some conditions on b, σ and f we have that the first order condition stated above uniquely determines α * for s ∈ [0, T ] by for some function ϕ. Therefore, after plugging (9) into (1) and (4), we obtain the following coupled FBSDE: where It is worth mentioning that when σ does not depend on the controlσ will depend on forward process and time only. This means thatσ does not have Y and Z components.
The theory of FBSDE has been studied widely and there are several methods to show the existence and uniqueness result, and a number of numerical algorithms have been proposed based on those methods. First is the method of contraction mapping. It was first studied by Antonelli [9] and later by Pardoux and Tang [15]. The main idea there is to show that a certain map is a contraction, and then to apply a fixed point argument. However, it turns out that this method works only for small enough time horizon T . In the case whenσ does not depend on Y and Z , having small T is sufficient to get contraction. Otherwise, one needs to assume additionally that the Lipschitz constants ofσ in z and that of g in x satisfy a certain inequality, see [26,Theorem 8.2.1]. Using the method of contraction mapping one can then implement a Picard-iterationtype numerical algorithm and show exponential convergence for small T . The second method is the Four Step Scheme. It was introduced by Ma et al., see [10], and was later studied by Delarue [17]. The idea is to use a decoupling function and then study an associated quasi-linear PDE. We note that in [10,17] the forward diffusion coefficient σ does not depend on Z . This corresponds to stochastic control problems with the uncontrolled diffusion coefficient. The numerical algorithms based on this method exploits the numerical solution of the associated quasi-linear PDE and therefore faces some limitations for high dimensional problems, see Douglas et al. [12], Milstein and Tretyakov [20], Ma et al. [21] and Delarue and Menozzi [18]. Guo et al. [24] proposed a numerical scheme for high-dimensional quasi-linear PDE associated with the coupled FBSDE whenσ does not depend on Z , which is based on a monotone scheme and on probabilistic approach. Finally, there is the method of continuation. This method was developed by Hu and Peng [11], Peng and Wu [16] and by Yong [14]. It allows them to show the existence and uniqueness result for arbitrary T under monotonicity conditions on the coefficients, which one would not expect to apply to FBSDEs arising from a control problem as described by (9), (10). Recently, deep learning methods have been applied to solving FBSDEs. In [35], three algorithms for solving fully coupled FBSDEs which have good accuracy and performance for high-dimensional problems are provided. One of the algorithms is based on the Picard iteration and it converges, but only for small enough T . In [34], an alternative algorithm for solving high-dimensional fully coupled FBSDEs based on deep learning was proposed, and the convergence result was shown assuming small T and other structural conditions (sometimes referred to as weak coupling and monotonicity conditions).

Main Results
We fix a finite horizon T ∈ (0, ∞). Let A be a separable metric space. This is the space where the control processes α take values. We fix a filtered probability space Wiener martingale on this space. By E t we denote the conditional expectation with respect to F t . Let | · | denote any norm in a finite dimensional Euclidean space. By · L ∞ we denote the norm in L ∞ ( ). Let Z H ∞ := ess sup (t,ω) |Z t (ω)| for any predictable process Z . We understand the following as D x l x n σ i j , where i, l, n = 1, 2, . . . , d and j = 1, 2, . . . , d . By Z we denote the transpose of Z .
The state of the system is governed by the controlled SDE (1). The corresponding adjoint equation satisfies (4).

Assumption 2.1
The functions b and σ are jointly continuous in t and twice differentiable in x. There exists K ≥ 0 such that ∀x ∈ R d , ∀a ∈ A, ∀t ∈ [0, T ], Moreover, assume that D 2 Clearly the assumption (11) implies that ∀x, x ∈ R d , ∀a ∈ A, ∀t ∈ [0, T ] we have The assumption that D 2 x σ (t, x, a) = 0 ∀x ∈ R d , ∀a ∈ A, ∀t ∈ [0, T ] is needed so that (21), in the proof of Lemma 3.1, holds. Without this assumption (21) would only hold if we could show that Z α H ∞ < ∞. Without additional regularization of the control problem this is impossible. Indeed, with [13, Proposition 5.3] we see that Z α t is a version of D t Y α t (the Malliavin derivative of Y α t ) and D t Y α t itself satisfies a linear BSDE. However, to obtain the estimates using this representation, one term that arises is D t α s where t ∈ [0, T ] and s ∈ [t, T ]. So we would need ess sup ω∈ ,t∈(0,T ),s∈(t,T ) |D t α s (ω)| < ∞. This is not necessarily the case here.

Assumption 2.2
The functions f is joinly continuous in t, and f and σ are twice differentiable in x. There is a constant K ≥ 0 such that ∀x, ∀a ∈ A, ∀t ∈ [0, T ] Under these assumptions, we can obtain the following estimate.

Lemma 2.3 Let Assumption 2.1 and 2.2 hold. Then for any admissible controls ϕ and
θ there exists a constant C > 0 such that The proof will be given in Sect. 3. We now state a necessary condition for optimality for the augmented Hamiltonian.

Theorem 2.4 [Extended Pontryagin's optimality principle]
Let α * be the (locally) optimal control, X α * be the associated controlled state solving (1), and (Y α * , Z α * ) be the associated adjoint processes solving (4). Then for any a ∈ A we havẽ The proof of Theorem 2.4 will come in Sect. 3. We are now ready to present the main result of the paper. Theorem 2.5 Let Assumptions 2.1 and 2.2 hold. Then Algorithm 1 converges to a local minimum of (2) for sufficiently large ρ > 0. Theorem 2.5 will be proved in Sect. 3. It can be seen from the proof that ρ needs to be two times larger than the constant appearing in Lemma 2.3, which itself depends increases with T , d and constants from Assumption 2.1 and 2.2.
We cannot guarantee that the Algorithm 1 converges to the optimal control which minimizes (2), since the extended Pontryagin's optimality principle, see Theorem 2.4, is the necessary condition for optimality. The sufficient condition for optimality tells us that to get the optimal control we need to assume convexity of the Hamiltonian in state and control variables, and need to assume convexity of the terminal cost function.
To that end, we need to assume convexity of b, σ, f and g in x and a.
In the following corollary, we show that under a particular setting of the problem we have logarithmic convergence of the modified method of successive approximations to the true solution of the problem.  f (t, x, a) = f 1 (t, x) + f 2 (t, a) for ∀t ∈ [0, T ], ∀x ∈ R d , ∀a ∈ A. In addition, assume that f and g are convex in x, f 2 , b 2 , σ 2 are convex in a. Then we have the following estimate for the sequence (α n ) n∈N from Algorithm 1: where α * is the optimal control for (2) and C is a positive constant.
The proof of Corollary 2.6 will be given in Sect. 3. Theorem 2.5 and Corollary 2.6 are extensions of the result in [5] to the stochastic case.

Proofs
We start working towards the proof of Theorem 2.5. Recall the adjoint equation for an admissible control α: From now on, we shall use Einstein notation, so that repeated indices in a single term imply summation over all the values of that index.

Lemma 3.1 Assume that there exists K
and Proof From the definition of the Hamiltonian (3) we have Hence, one can observe that (16) is a linear BSDE. Therefore, from [33, Proposition 3.2] we can write the formula for the solution of (16): where the process S is the unique strong solution of and S −1 is the inverse process of S. Thus, due to [33,Corollary 3.7] and assumptions of lemma we have the following bound: Hence, due to assumptions of lemma we conclude that Y α H ∞ is bounded.
Proof of Lemma 2.3 Let ϕ and θ be some generic admissible controls. We will write (X ϕ s ) s∈[0,T ] for the solution of (1) controlled by ϕ and (X θ s ) s∈[0,T ] for the solution of (1) controlled by θ . We denote solutions of corresponding adjoint equations by Due to Taylor's theorem, we note that for some The last inequality holds due to Assumption 2.2. Recall that Y θ T = D x g(X θ T ). Hence, using Itô's product rule, we get From this, the forward SDE (1) and the adjoint equation (4) we thus get On the other hand, by definition of the Hamiltonian we have Summing up (17) and (18) we get Since Therefore, after substituting (20) into (19), and by 21 we get Let us now get a standard SDE estimate for the difference of X ϕ and X θ . From (a + b) 2 ≤ 2a 2 + 2b 2 , from taking the expectation, from Hölder's inequality, from Assumption 2.1, from the Burkholder-Davis-Gundy inequality and from Gronwall's inequality we obtain Young's inequality allows us to get the estimate Hence, from (22) Since (−μ(α n−1 )) ≥ 0 and ∞ n=1 (−μ(α n−1 )) < +∞ we have that μ(α n−1 ) → 0 as n → 0. This concludes the proof.
We need to introduce new notation, which will be used in the proof of Corollary 2.6. Denote the set Hence, by (28) we get Last inequality holds since h * N (h * ) T is less or equal to 1. Hence we get the contradiction. ) − f (s, X s , α * s ) ds by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.