The classical PDE approach to dynamic programming

Abstract

In this chapter, we use the dynamic programming method for solving stochastic control problems. We consider in Section 3.2 the framework of controlled diffusion and the problem is formulated on finite or infinite horizon. The basic idea of the approach is to consider a family of control problems by varying the initial state values, and to derive some relations between the associated value functions. This principle, called the dynamic programming principle and initiated in the 1950s by Bellman, is stated precisely in Section 3.3. This approach yields a certain partial differential equation (PDE), of second order and nonlinear, called Hamilton-Jacobi-Bellman (HJB), and formally derived in Section 3.4. When this PDE can be solved by the explicit or theoretical achievement of a smooth solution, the verification theorem proved in Section 3.5, validates the optimality of the candidate solution to the HJB equation. This classical approach to the dynamic programming is called the verification step. We illustrate this method in Section 3.6 by solving three examples in finance. The main drawback of this approach is to suppose the existence of a regular solution to the HJB equation. This is not the case in general, and we give in Section 3.7 a simple example inspired by finance pointing out this feature.