Particle methods for stochastic optimal control problems

Abstract

When dealing with numerical solution of stochastic optimal control problems, stochastic dynamic programming is the natural framework. In order to try to overcome the so-called curse of dimensionality, the stochastic programming school promoted another approach based on scenario trees which can be seen as the combination of Monte Carlo sampling ideas on the one hand, and of a heuristic technique to handle causality (or nonanticipativeness) constraints on the other hand.

However, if one considers that the solution of a stochastic optimal control problem is a feedback law which relates control to state variables, the numerical resolution of the optimization problem over a scenario tree should be completed by a feedback synthesis stage in which, at each time step of the scenario tree, control values at nodes are plotted against corresponding state values to provide a first discrete shape of this feedback law from which a continuous function can be finally inferred. From this point of view, the scenario tree approach faces an important difficulty: at the first time stages (close to the tree root), there are a few nodes (or Monte-Carlo particles), and therefore a relatively scarce amount of information to guess a feedback law, but this information is generally of a good quality (that is, viewed as a set of control value estimates for some particular state values, it has a small variance because the future of those nodes is rich enough); on the contrary, at the final time stages (near the tree leaves), the number of nodes increases but the variance gets large because the future of each node gets poor (and sometimes even deterministic).

After this dilemma has been confirmed by numerical experiments, we have tried to derive new variational approaches. First of all, two different formulations of the essential constraint of nonanticipativeness are considered: one is called algebraic and the other one is called functional. Next, in both settings, we obtain optimality conditions for the corresponding optimal control problem. For the numerical resolution of those optimality conditions, an adaptive mesh discretization method is used in the state space in order to provide information for feedback synthesis. This mesh is naturally derived from a bunch of sample noise trajectories which need not to be put into the form of a tree prior to numerical resolution. In particular, an important consequence of this discrepancy with the scenario tree approach is that the same number of nodes (or points) are available from the beginning to the end of the time horizon. And this will be obtained without sacrifying the quality of the results (that is, the variance of the estimates). Results of experiments with a hydro-electric dam production management problem will be presented and will demonstrate the claimed improvements. A more realistic problem will also be presented in order to demonstrate the effectiveness of the method for high dimensional problems.

Appendix: Optimization on an Hilbert space: a special case

Let $\mathcal {H}$ be an Hilbert space, let $\mathcal {H}^{\mathrm{fe}}$ be a closed convex subset of $\mathcal {H}$ and let f be a real valued function defined on $\mathcal {H}$. We consider the following optimization problem:

$$ \min_{x\in \mathcal {H}^{\mathrm{fe}}} \ f(x) . $$

(33)

In the following, $\chi _{{}_{H}}$ will denote the indicator function of a subset $H\subset \mathcal {H}$, namely

$$ \chi _{{}_{H}}(x) = \left \{ \begin{array}{l@{\quad}l} 0 & \text{if $x\in H$} ,\\ +\infty & \text{otherwise} .\end{array} \right . $$

The optimization literature gives different expressions for the necessary optimality conditions of an optimization problem in a general Hilbert space (see e.g. [13]). For instance, if $x^{\sharp}\in \mathcal {H}$ is solution of (33), then the following statements are equivalent:

$$\begin{aligned} & \forall x\in \mathcal {H}^{\mathrm{fe}},\quad \big \langle f'\bigl(x^\sharp \bigr),x-x^\sharp\big \rangle \geq 0 , \end{aligned}$$

(34a)

$$\begin{aligned} & f'\bigl(x^\sharp\bigr)\in -\partial \chi _{{}_{ \mathcal {H}^{\mathrm{fe}}}} \bigl(x^\sharp\bigr) , \end{aligned}$$

(34b)

$$\begin{aligned} & \forall \epsilon> 0 ,x^\sharp= \mathrm {proj}_{ \mathcal {H}^{\mathrm{fe}}}\left (x^\sharp- \epsilon \nabla _{}f\bigl(x^\sharp\bigr)\right ) . \end{aligned}$$

(34c)

We now consider a specific structure for the feasible set $\mathcal {H}^{\mathrm{fe}}$. More precisely, we assume that $\mathcal {H}^{\mathrm{fe}}= \mathcal {H}^{\mathrm{cv}}\cap \mathcal {H}^{\mathrm{sp}}$, $\mathcal {H}^{\mathrm{sp}}$ being a closed subspace of $\mathcal {H}$ and $\mathcal {H}^{\mathrm{cv}}$ being a closed convex subset of $\mathcal {H}$. We moreover assume that the following property holds.

Assumption 5

The sets $\mathcal {H}^{\mathrm{sp}}$ and $\mathcal {H}^{\mathrm{cv}}$ are such that $\mathrm {proj}_{ \mathcal {H}^{\mathrm{cv}}}\left ( \mathcal {H}^{\mathrm{sp}}\right )\subset \mathcal {H}^{\mathrm{sp}}$.

Then the projection operator on $\mathcal {H}^{\mathrm{fe}}$ has the following property.

Lemma 2

Under Assumption 5, the following relation holds true:

$$ \mathrm {proj}_{ \mathcal {H}^{\mathrm{cv}}\cap \mathcal {H}^{\mathrm{sp}}}=\mathrm {proj}_{ \mathcal {H}^{\mathrm{cv}}}\circ \mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}} .$$

Proof

Let $y\in \mathcal {H}^{\mathrm{cv}}\cap \mathcal {H}^{\mathrm{sp}}$. Then

$$\begin{aligned} &\big \langle x- \mathrm {proj}_{ \mathcal {H}^{\mathrm{cv}}}\left ( \mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}}\left (x\right )\right ),y- \mathrm {proj}_{ \mathcal {H}^{\mathrm{cv}}}\left ( \mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}}\left (x\right )\right )\big \rangle \\ &\quad = \big \langle x- \mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}}\left (x\right ),y- \mathrm {proj}_{ \mathcal {H}^{\mathrm{cv}}}\left ( \mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}}\left (x\right )\right )\big \rangle \\ &\qquad{} + \big \langle \mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}}\left (x\right )- \mathrm {proj}_{ \mathcal {H}^{\mathrm{cv}}}\left ( \mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}}\left (x\right )\right ), y- \mathrm {proj}_{ \mathcal {H}^{\mathrm{cv}}}\left ( \mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}}\left (x\right )\right )\big \rangle . \end{aligned}$$

From the characterization of the projection of $z:= \mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}}\left (x\right )$ over the convex subset $\mathcal {H}^{\mathrm{cv}}$, the last inner product in the previous expression is non positive: therefore,

$$ \big \langle x- \mathrm {proj}_{ \mathcal {H}^{\mathrm{cv}}}\left (z\right ),y- \mathrm {proj}_{ \mathcal {H}^{\mathrm{cv}}}\left (z\right )\big \rangle \leq \big \langle x- z,y- \mathrm {proj}_{ \mathcal {H}^{\mathrm{cv}}}\left (z\right )\big \rangle .$$

From $\mathrm {proj}_{ \mathcal {H}^{\mathrm{cv}}}\left ( \mathcal {H}^{\mathrm{sp}}\right )\subset \mathcal {H}^{\mathrm{sp}}$, we deduce that $y- \mathrm {proj}_{ \mathcal {H}^{\mathrm{cv}}}\left (z\right )\in \mathcal {H}^{\mathrm{sp}}$. Since $\mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}}$ is a self-adjoint operator, we have

$$\begin{aligned} \big \langle x- \mathrm {proj}_{ \mathcal {H}^{\mathrm{cv}}}\left (z\right ),y- \mathrm {proj}_{ \mathcal {H}^{\mathrm{cv}}}\left (z\right )\big \rangle &\leq \big \langle x- z, \mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}}\left (y- \mathrm {proj}_{ \mathcal {H}^{\mathrm{cv}}}\left (z\right )\right )\big \rangle ,\\ &\leq \big \langle \mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}}\left (x-z\right ),y- \mathrm {proj}_{ \mathcal {H}^{\mathrm{cv}}}\left (z\right )\big \rangle ,\\ &\leq 0 , \end{aligned}$$

the last inequality arising from $\mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}}\left (x-z\right ) = \mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}}\left (x\right )- \mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}}\left (z\right ) =0$ (since $\mathcal {H}^{\mathrm{sp}}$ is a linear subspace, then $\mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}}\left (\cdot\right )$ is a linear operator). We thus conclude that, for all $y\in \mathcal {H}^{\mathrm{cv}}\cap \mathcal {H}^{\mathrm{sp}}$,

$$ \big \langle x-\mathrm {proj}_{ \mathcal {H}^{\mathrm{cv}}}\circ \mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}}\left (x\right ),y- \mathrm {proj}_{ \mathcal {H}^{\mathrm{cv}}}\circ \mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}}\left (x\right )\big \rangle \leq 0 ,$$

a variational inequality which characterizes $\mathrm {proj}_{ \mathcal {H}^{\mathrm{cv}}}\circ \mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}}\left (x\right )$ as the projection of x over $\mathcal {H}^{\mathrm{fe}}= \mathcal {H}^{\mathrm{cv}}\cap \mathcal {H}^{\mathrm{sp}}$.    □

The following proposition gives necessary optimality conditions for Problem (33) when the feasible set $\mathcal {H}^{\mathrm{fe}}$ has the specific structure $\mathcal {H}^{\mathrm{cv}}\cap \mathcal {H}^{\mathrm{sp}}$.

Proposition 5

We suppose that Assumption 5 is fulfilled and that f is differentiable. If x ^♯ is solution of (33), then

$$ \mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}}\left (f'\bigl(x^\sharp\bigr)\right )\in -\partial \chi _{{}_{ \mathcal {H}^{\mathrm{cv}}}}\bigl(x^\sharp\bigr) .$$

Proof

Let x ^♯ be solution of (33). Using Condition (34c) and Lemma 2, we obtain that

$$ x^\sharp=\mathrm {proj}_{ \mathcal {H}^{\mathrm{cv}}}\circ \mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}}\left (x^\sharp-\epsilon \nabla _{}f\bigl(x^\sharp\bigr)\right ),\quad \forall \epsilon \geq 0 .$$

But $\mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}}$ is a linear operator and $x^{\sharp}\in \mathcal {H}^{\mathrm{sp}}$, so that

$$ x^\sharp= \mathrm {proj}_{ \mathcal {H}^{\mathrm{cv}}}\left (x^\sharp-\epsilon\: \mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}}\left (\nabla _{}f \bigl(x^\sharp\bigr)\right )\right ) .$$

From (34a)–(34c), the last relation is equivalent to $\mathrm {proj}_{ \mathcal {H}^{\mathrm{sp}}}(f'(x^{\sharp}))\in -\partial \chi _{{}_{ \mathcal {H}^{\mathrm{cv}}}}(x^{\sharp})$. □