Applied Mathematics and Optimization

, Volume 57, Issue 2, pp 177–206 | Cite as

Time Discretisation and Rate of Convergence for the Optimal Control of Continuous-Time Stochastic Systems with Delay

  • Markus FischerEmail author
  • Giovanna Nappo


We study a semi-discretisation scheme for stochastic optimal control problems whose dynamics are given by controlled stochastic delay (or functional) differential equations with bounded memory. Performance is measured in terms of expected costs. By discretising time in two steps, we construct a sequence of approximating finite-dimensional Markovian optimal control problems in discrete time. The corresponding value functions converge to the value function of the original problem, and we derive an upper bound on the discretisation error or, equivalently, a worst-case estimate for the rate of convergence.


Optimal control Stochastic differential equation Functional differential equation Delay Time lag Finite differences Time discretisation Approximation Error bound Convergence rate 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Atkinson, K., Han, W.: Theoretical Numerical Analysis. Texts in Applied Mathematics, vol. 39. Springer, New York (2001) zbMATHGoogle Scholar
  2. 2.
    Barles, G., Jakobsen, E.R.: Error bounds for monotone approximation schemes for Hamilton-Jacobi-Bellman equations. SIAM J. Numer. Anal. 43(2), 540–558 (2005) zbMATHCrossRefMathSciNetGoogle Scholar
  3. 3.
    Barles, G., Jakobsen, E.R.: Error bounds for monotone approximation schemes for parabolic Hamilton-Jacobi-Bellman equations. Math. Comput. 76(260), 1861–1893 (2007) zbMATHCrossRefMathSciNetGoogle Scholar
  4. 4.
    Bauer, H., Rieder, U.: Stochastic control problems with delay. Math. Meth. Oper. Res. 62(3), 411–427 (2005) zbMATHCrossRefMathSciNetGoogle Scholar
  5. 5.
    Buckwar, E.: Introduction to the numerical analysis of stochastic delay differential equations. J. Comput. Appl. Math. 125(1–2), 297–307 (2000) zbMATHCrossRefMathSciNetGoogle Scholar
  6. 6.
    Calzolari, A., Florchinger, P., Nappo, G.: Convergence in nonlinear filtering for stochastic delay systems. SIAM J. Control Optim. (to appear) Google Scholar
  7. 7.
    Chang, M.-H., Pang, T., Pemy, M.: Optimal control of stochastic functional differential equations with bounded memory. In: Zhang, X., Liu, D., Wu, L. (eds.) Operations Research and Its Applications, Papers from the Sixth International Symposium, ISORA’06, Xinjiang, China, August 8–12, pp. 82–94. World Publishing Corporation, Beijing (2006) Google Scholar
  8. 8.
    Elsanosi, I., Øksendal, B., Sulem, A.: Some solvable stochastic control problems with delay. Stoch. Stoch. Rep. 71(1–2), 69–89 (2000) zbMATHGoogle Scholar
  9. 9.
    Falcone, M., Ferretti, R.: Discrete time high-order schemes for viscosity solutions of Hamilton-Jacobi-Bellman equations. Numer. Math. 67(3), 315–344 (1994) zbMATHCrossRefMathSciNetGoogle Scholar
  10. 10.
    Falcone, M., Rosace, R.: Discrete-time approximation of optimal control problems for delayed equations. Control Cybern. 25(3), 665–675 (1996) zbMATHMathSciNetGoogle Scholar
  11. 11.
    Fischer, M., Reiß, M.: Discretisation of stochastic control problems for continuous time dynamics with delay. J. Comput. Appl. Math. 205(2), 969–981 (2007) zbMATHCrossRefMathSciNetGoogle Scholar
  12. 12.
    Garsia, A.M., Rodemich, E., Rumsey, H. Jr.: A real variable lemma and the continuity of paths of some Gaussian processes. Indiana Math. J. 20(6), 565–578 (1970) zbMATHCrossRefMathSciNetGoogle Scholar
  13. 13.
    Gihman, I.I., Skorohod, A.V.: Controlled Stochastic Processes. Springer, New York (1979) Google Scholar
  14. 14.
    Hu, Y., Mohammed, S.-E.A., Yan, F.: Discrete-time approximations of stochastic delay equations: the Milstein scheme. Ann. Probab. 32(1A), 265–314 (2004) zbMATHCrossRefMathSciNetGoogle Scholar
  15. 15.
    Karatzas, I., Shreve, S.E.: Brownian Motion and Stochastic Calculus, 2nd edn. Graduate Texts in Mathematics, vol. 113. Springer, New York (1991) zbMATHGoogle Scholar
  16. 16.
    Krylov, N.V.: Controlled Diffusion Processes. Applications of Mathematics, vol. 14. Springer, New York (1980) zbMATHGoogle Scholar
  17. 17.
    Krylov, N.V.: Approximating value functions for controlled degenerate diffusion processes by using piece-wise constant policies. Electron. J. Probab. 4, 1–19 (1999) MathSciNetGoogle Scholar
  18. 18.
    Krylov, N.V.: On the rate of convergence of finite-difference approximations for Bellman’s equations with variable coefficients. Probab. Theory Relat. Fields 117(1), 1–16 (2000) zbMATHCrossRefGoogle Scholar
  19. 19.
    Krylov, N.V.: Mean value theorems for stochastic integrals. Ann. Probab. 29(1), 385–410 (2001) zbMATHCrossRefMathSciNetGoogle Scholar
  20. 20.
    Kushner, H.J.: Numerical approximations for nonlinear stochastic systems with delays. Stochastics 77(3), 211–240 (2005) zbMATHMathSciNetGoogle Scholar
  21. 21.
    Kushner, H.J.: Numerical approximations for stochastic systems with delays in the state and control. Technical Report, Lefschetz Center for Dynamical systems (2005) Google Scholar
  22. 22.
    Kushner, H.J., Dupuis, P.: Numerical Methods for Stochastic Control Problems in Continuous Time, 2nd edn. Applications of Mathematics, vol. 24. Springer, New York (2001) zbMATHGoogle Scholar
  23. 23.
    Larssen, B.: Dynamic programming in stochastic control of systems with delay. Stoch. Stoch. Rep. 74(3–4), 651–673 (2002) zbMATHCrossRefMathSciNetGoogle Scholar
  24. 24.
    Larssen, B., Risebro, N.H.: When are HJB-equations for control problems with stochastic delay equations finite dimensional? Stoch. Anal. Appl. 21(3), 643–671 (2003) zbMATHCrossRefMathSciNetGoogle Scholar
  25. 25.
    Mao, X.: Numerical solutions of stochastic functional differential equations. LMS J. Comput. Math. 6, 141–161 (2003) zbMATHMathSciNetGoogle Scholar
  26. 26.
    Mohammed, S.-E.A.: Stochastic Functional Differential Equations. Pitman, London (1984) zbMATHGoogle Scholar
  27. 27.
    Øksendal, B., Sulem, A.: A maximum principle for optimal control of stochastic systems with delay, with applications to finance. In: Menaldi, J., Rofman, E., Sulem, A. (eds.) Optimal Control and Partial Differential Equations. In Honour of Professor Alain Bensoussan’s 60th Birthday. Proceedings of the Conference, Paris, December 4, 2000, pp. 64–79. IOS Press, Amsterdam (2001) Google Scholar
  28. 28.
    Rogers, L.C.G.: Pathwise stochastic optimal control. SIAM J. Control Optim. 46(3), 1116–1132 (2007) CrossRefMathSciNetGoogle Scholar
  29. 29.
    Słomiński, L.: Euler’s approximations of solutions of SDEs with reflecting boundary. Stoch. Process. Appl. 94, 317–337 (2001) CrossRefGoogle Scholar
  30. 30.
    Stroock, D.W., Varadhan, S.R.S.: Multidimensional Diffusion Processes. Grundlehren der Mathematischen Wissenschaften, vol. 233. Springer, Berlin (1979) zbMATHGoogle Scholar
  31. 31.
    Yong, J., Zhou, X.Y.: Stochastic Controls. Hamiltonian Systems and HJB Equations. Applications of Mathematics, vol. 43. Springer, New York (1999) zbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2007

Authors and Affiliations

  1. 1.Institute for Applied MathematicsUniversity of HeidelbergHeidelbergGermany
  2. 2.Department of MathematicsUniversity of Rome “La Sapienza”RomeItaly

Personalised recommendations