Abstract
We propose a simple time-discretization scheme for multi-dimensional stochastic optimal control problems in continuous time. It is based on a probabilistic representation for the convolution of the value function by a probability density function. We show the convergence results under mild conditions on coefficients of the problems by Barles–Souganidis viscosity solution method. Resulting numerical methods allow us to use uncontrolled Markov processes to estimate the conditional expectations in the dynamic programming procedure. Moreover, it can be implemented without the interpolation of the value function or the adjustment of the diffusion matrix.
Similar content being viewed by others
References
Barles, G., Jakobsen, E.R.: On the convergence rate of approximation schemes for Hamilton–Jacobi–Bellman equations. Math. Model. Numer. Anal. 36, 33–54 (2002)
Barles, G., Jakobsen, E.R.: Error bounds for monotone approximation schemes for parabolic Hamilton–Jacobi–Bellman equations. Math. Comp. 76, 1861–1893 (2007)
Barles, G., Souganidis, P.E.: Convergence of approximation schemes for fully nonlinear second order equations. Asymptot. Anal. 4, 271–283 (1991)
Bonnans, F., Ottenwaelter, E., Zidani, H.: A fast algorithm for the two dimensional HJB equation of stochastic control. Math. Model. Numer. Anal. 38, 723–735 (2004)
Bonnans, F., Zidani, H.: Consistency of generalized finite difference schemes for the stochastic HJB equation. SIAM J. Numer. Anal. 41, 1008–1021 (2003)
Bouchard, B., Touzi, N.: Discrete-time approximation and Monte Carlo simulation of backward stochastic differential equations. Stoch. Process. Appl. 111, 175–206 (2004)
Camilli, F., Falcone, M.: An approximation scheme for the optimal control of diffusion processes. Math. Model. Numer. Anal. 29, 97–122 (1995)
Carlini, E., Falcone, M., Ferretti, R.: An efficient algorithm for Hamilton–Jacobi equations in high dimension. Comput. Vis. Sci. 7, 15–29 (2004)
Debrabant, K., Jakobsen, E.R.: Semi-Lagrangian schemes for linear and fully non-linear diffusion equations. Math. Comput. 82, 1433–1462 (2013)
Fahim, A., Touzi, N., Warin, X.: A probabilistic numerical method for fully nonlinear parabolic PDEs. Ann. Appl. Probab. 21, 1322–1364 (2011)
Fleming, W.H., Soner, H.M.: Controlled Markov Processes and Viscosity Solutions, 2nd edn. Springer, New York (2006)
Graf, S., Luschgy, H.: Foundation of Quantization for Probability Distributions. Springer, Berlin (2000)
Györfi, L., Kohler, M., Krzyzak, A., Walk, H.: A distribution-free theory of nonparametric regression. Springer, New York (2002)
Kaise, H., McEneaney, W.M.: Idempotent expansions for continuous-time stochastic control. In: Proceedings of 49th IEEE conference on decision and control (2010)
Kohn, R.V., Serfaty, S.: A deterministic-control-based approach to fully nonlinear parabolic and elliptic equations. Commun. Pure Appl. Math. 63, 1298–1350 (2010)
Krylov, N.V.: Controlled Diffusion Processes. Springer, New York (1980)
Krylov, N.V.: Approximating value functions for controlled degenerate diffusion processes by using piece-wise constant policies. Electron. J. Probab. 4, 1–19 (1999)
Kushner, H.J., Dupuis, P.: Numerical Methods for Stochastic Control Problems in Continuous Time. Springer, New York (2001)
Lemor, J.P., Gobet, E., Warin, X.: Rate of convergence of an empirical regression method for solving generalized backward stochastic differential equations. Bernoulli 12, 889–916 (2006)
Longstaff, F.A., Schwartz, E.S.: Valuing american options by simulation: a simple least-squares approach. Rev. Financ. Stud. 14, 113–147 (2001)
Pagès, G., Pham, H., Printems, J.: An optimal Markovian quantization algorithm for multidimensional stochastic control problems. Stoch. Dyn. 4, 501–545 (2004)
Pagès, G., Printems, J.: Optimal quadratic quantization for numerics: the Gaussian case. Monte Carlo Methods Appl. 9, 135–168 (2003)
Pham, H.: Continuous-Time Stochastic Control and Optimization With Financial Applications. Springer, Berlin (2009)
Acknowledgments
The author is thankful to Takashi Tamura for helpful comments, as well as the anonymous referee for his/her careful reading of a previous version of this paper.
Author information
Authors and Affiliations
Corresponding author
Additional information
This study is partially supported by JSPS KAKENHI Grant Number 26800079.
About this article
Cite this article
Nakano, Y. An approximation scheme for stochastic controls in continuous time. Japan J. Indust. Appl. Math. 31, 681–696 (2014). https://doi.org/10.1007/s13160-014-0157-1
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13160-014-0157-1
Keywords
- Stochastic controls
- Hamilton–Jacobi–Bellman equations
- Viscosity solutions
- Kernel density estimation
- Monte Carlo methods