Encyclopedia of Systems and Control

2015 Edition
| Editors: John Baillieul, Tariq Samad

Numerical Methods for Continuous-Time Stochastic Control Problems

  • George Yin
Reference work entry
DOI: https://doi.org/10.1007/978-1-4471-5058-9_232


This expository article provides a brief review of numerical methods for stochastic control in continuous time. It concentrates on the methods of Markov chain approximation for controlled diffusions. Leaving most of the technical details out with the broad general audience in mind, it aims to serve as an introductory reference or a user’s guide for researchers, practitioners, and students who wish to know something about numerical methods for stochastic control.


Markov chain approximation Numerical methods Stochastic control 
This is a preview of subscription content, log in to check access.



The research of this author was supported in part by the Army Research Office under grant W911NF-12-1-0223.


  1. Barles G, Souganidis P (1991) Convergence of approximation schemes for fully nonlinear second order equations. J Asymptot Anal 4:271–283MathSciNetGoogle Scholar
  2. Bertsekas DP, Castanon DA (1989) Adaptive aggregation methods for infinite horizon dynamic programming. IEEE Trans Autom Control 34:589–598MathSciNetGoogle Scholar
  3. Chancelier P, Gomez C, Quadrat J-P, Sulem A, Blankenship GL, La Vigna A, MaCenary DC, Yan I (1986) An expert system for control and signal processing with automatic FORTRAN program generation. In: Mathematical systems symposium, Stockholm. Royal Institute of Technology, StockholmGoogle Scholar
  4. Chancelier P, Gomez C, Quadrat J-P, Sulem A (1987) Automatic study in stochastic control. In: Fleming W, Lions PL (eds) IMA volume in mathematics and its applications, vol 10. Springer, BerlinGoogle Scholar
  5. Crandall MG, Lions PL (1983) Viscosity solutions of Hamilton-Jacobi equations. Trans Am Math Soc 277:1–42MathSciNetGoogle Scholar
  6. Crandall MG, Ishii H, Lions PL (1992) User’s guide to viscosity solutions of second order partial differential equations. Bull Am Math Soc 27:1–67MathSciNetGoogle Scholar
  7. Fleming WH, Rishel RW (1975) Deterministic and stochastic optimal control. Springer, New YorkGoogle Scholar
  8. Fleming WH, Soner HM (1992) Controlled Markov processes and viscosity solutions. Springer, New YorkGoogle Scholar
  9. Jin Z, Wang Y, Yin G (2011) Numerical solutions of quantile hedging for guaranteed minimum death benefits under a regime-switching-jump-diffusion formulation. J Comput Appl Math 235:2842–2860MathSciNetGoogle Scholar
  10. Jin Z, Yin G, Zhu C (2012) Numerical solutions of optimal risk control and dividend optimization policies under a generalized singular control formulation. Automatica 48:1489–1501MathSciNetGoogle Scholar
  11. Jin Z, Yang HL, Yin G (2013) Numerical methods for optimal dividend payment and investment strategies of regime-switching jump diffusion models with capital injections. Automatica 49:2317–2329MathSciNetGoogle Scholar
  12. Krylov VN (2000) On the rate of convergence of finite-difference approximations for Bellman’s equations with variable coefficients. Probab Theory Relat Fields 117:1–16Google Scholar
  13. Kushner HJ (1977) Probability methods for approximation in stochastic control and for elliptic equations. Academic, New YorkGoogle Scholar
  14. Kushner HJ (1990a) Weak convergence methods and singularly perturbed stochastic control and filtering problems. Birkhäuser, BostonGoogle Scholar
  15. Kushner HJ (1990b) Numiercal methods for stochastic control problems in continuous time. SIAM J Control Optim 28:999–1048MathSciNetGoogle Scholar
  16. Kushner HJ (2000) Consistency issues for numerical methods for variance control with applications to optimization in finance. IEEE Trans Autom Control 44:2283–2296MathSciNetGoogle Scholar
  17. Kushner HJ (2002) Numerical approximations for stochastic differential games. SIAM J Control Optim 40:457–486MathSciNetGoogle Scholar
  18. Kushner HJ (2008) Numerical methods for controlled stochastic delay systems. Birkhäuser, BostonGoogle Scholar
  19. Kushner HJ, Dupuis PG (1992) Numerical methods for stochastic control problems in continuous time. Springer, New YorkGoogle Scholar
  20. Kushner HJ, Kleinman AJ (1968) Numerical methods for the solution of the degenerate nonlinear elliptic equations arising in optimal stochastic control theory. IEEE Trans Autom Control AC-13:344–353MathSciNetGoogle Scholar
  21. Kushner HJ, Yin G (2003) Stochastic approximation and recursive algorithms and applications, 2nd edn. Springer, New YorkGoogle Scholar
  22. Menaldi J (1989) Some estimates for finite difference approximations. SIAM J Control Optim 27:579–607MathSciNetGoogle Scholar
  23. Peng S (1990) A general stochastic maximum principle for optimal control problems. SIAM J Control Optim 28:966–979MathSciNetGoogle Scholar
  24. Sethi SP, Zhang Q (1994) Hierarchical decision making in stochastic manufacturing systems. Birkhäuser, BostonGoogle Scholar
  25. Song QS, Yin G (2009) Rates of convergence of numerical methods for controlled regime-switching diffusions with stopping times in the costs. SIAM J Control Optim 48:1831–1857MathSciNetGoogle Scholar
  26. Song QS, Yin G, Zhang Z (2006) Numerical method for controlled regime-switching diffusions and regime-switching jump diffusions. Automatica 42:1147–1157MathSciNetGoogle Scholar
  27. Song QS, Yin G, Zhang Z (2008) Numerical solutions for stochastic differential games with regime switching. IEEE Trans Autom Control 53:509–521MathSciNetGoogle Scholar
  28. Warga J (1962) Relaxed variational problems. J Math Anal Appl 4:111–128MathSciNetGoogle Scholar
  29. Yan HM, Yin G, Lou SXC (1994) Using stochastic optimization to determine threshold values for control of unreliable manufacturing systems. J Optim Theory Appl 83:511–539MathSciNetGoogle Scholar
  30. Yin G, Zhang Q (2013) Continuous-time Markov chains and applications: a two-time-scale approach, 2nd edn. Springer, New YorkGoogle Scholar
  31. Yin G, Zhu C (2010) Hybrid switching diffusions: properties and applications. Springer, New YorkGoogle Scholar
  32. Yin G, Liu RH, Zhang Q (2002) Recursive algorithms for stock liquidation: a stochastic optimization approach. SIAM J Optim 13:240–263MathSciNetGoogle Scholar
  33. Yin G, Jin H, Jin Z (2009) Numerical methods for portfolio selection with bounded constraints. J Comput Appl Math 233:564–581MathSciNetGoogle Scholar
  34. Yong J, Zhou XY (1999) Stochastic controls: Hamiltonian systems and HJB equations. Springer, New YorkGoogle Scholar
  35. Zhang J (2006) Rate of convergence of finite difference approximations for degenerate ODEs. Math Comput 75:1755–1778Google Scholar

Copyright information

© Springer-Verlag London 2015

Authors and Affiliations

  • George Yin
    • 1
  1. 1.Department of MathematicsWayne State UniversityDetroitUSA