Encyclopedia of Systems and Control

Living Edition
| Editors: John Baillieul, Tariq Samad

Numerical Methods for Continuous-Time Stochastic Control Problems

Living reference work entry
DOI: https://doi.org/10.1007/978-1-4471-5102-9_232-1

Introduction

This expository article provides a brief review of numerical methods for stochastic control in continuous time. Leaving most of the technical details out with the broad general audience in mind, it aims to serve as an introductory reference for researchers, practitioners, and students, who wish to know something about numerical methods for stochastic controls.

The study of stochastic control has witnessed tremendous progress in the last few decades; see, for example, Fleming and Rishel (1975), Fleming and Soner (1992), Kushner (1977), and Yong and Zhou (1999) among others, for fundamentals of stochastic controls as well as historical remarks. Much of the development has been accompanied by the needs and progress in science, engineering, as well as finance. Typically, the problems are highly nonlinear, so a closed-form solution is very difficult to obtain. As a result, designing feasible numerical algorithms becomes vitally important. Among the many approximation methods,...

Keywords

Hedging 
This is a preview of subscription content, log in to check access

Notes

Acknowledgements

Research of this author was supported in part by the Army Research Office under grant W911NF-12-1-0223.

Bibliography

  1. Barles G, Souganidis P (1991) Convergence of approximation schemes for fully nonlinear second order equations. J Asymptot Anal 4:271–283MATHMathSciNetGoogle Scholar
  2. Bertsekas DP, Castanon DA (1989) Adaptive aggregation methods for infinite horizon dynamic programming. IEEE Trans Autom Control 34:589–598CrossRefMATHMathSciNetGoogle Scholar
  3. Chancelier P, Gomez C, Quadrat J-P, Sulem A, Blankenship GL, La Vigna A, MaCenary DC, Yan I (1986) An expert system for control and signal processing with automatic FORTRAN program generation. In: Mathematical systems symposium, Stockholm. Royal Institute of Technology, StockholmGoogle Scholar
  4. Chancelier P, Gomez C, Quadrat J-P, Sulem A (1987) Automatic study in stochastic control. In: Fleming W, Lions PL (eds) IMA volume in mathematics and its applications, vol 10. Springer, BerlinGoogle Scholar
  5. Crandall MG, Lions PL (1983) Viscosity solutions of Hamilton-Jacobi equations. Trans Am Math Soc 277:1–42CrossRefMATHMathSciNetGoogle Scholar
  6. Crandall MG, Ishii H, Lions PL (1992) User’s guide to viscosity solutions of second order partial differential equations. Bull Am Math Soc 27:1–67CrossRefMATHMathSciNetGoogle Scholar
  7. Fleming WH, Rishel RW (1975) Deterministic and stochastic optimal control. Springer, New YorkCrossRefMATHGoogle Scholar
  8. Fleming WH, Soner HM (1992) Controlled Markov processes and viscosity solutions. Springer, New YorkGoogle Scholar
  9. Jin Z, Wang Y, Yin G (2011) Numerical solutions of quantile hedging for guaranteed minimum death benefits under a regime-switching-jump-diffusion formulation. J Comput Appl Math 235:2842–2860CrossRefMATHMathSciNetGoogle Scholar
  10. Jin Z, Yin G, Zhu C (2012) Numerical solutions of optimal risk control and dividend optimization policies under a generalized singular control formulation. Automatica 48:1489–1501CrossRefMATHMathSciNetGoogle Scholar
  11. Jin Z, Yang HL, Yin G (2013) Numerical methods for optimal dividend payment and investment strategies of regime-switching jump diffusion models with capital injections. Automatica 49:2317–2329CrossRefMathSciNetGoogle Scholar
  12. Krylov VN (2000) On the rate of convergence of finite-difference approximations for Bellman’s equations with variable coefficients. Probab Theory Relat Fields 117:1–16CrossRefMATHGoogle Scholar
  13. Kushner HJ (1977) Probability methods for approximation in stochastic control and for elliptic equations. Academic, New YorkGoogle Scholar
  14. Kushner HJ (1990a) Weak convergence methods and singularly perturbed stochastic control and filtering problems. Birkhäuser, BostonCrossRefMATHGoogle Scholar
  15. Kushner HJ (1990b) Numiercal methods for stochastic control problems in continuous time. SIAM J Control Optim 28:999–1048CrossRefMATHMathSciNetGoogle Scholar
  16. Kushner HJ (2000) Consistency issues for numerical methods for variance control with applications to optimization in finance. IEEE Trans Autom Control 44:2283–2296CrossRefMathSciNetGoogle Scholar
  17. Kushner HJ (2002) Numerical approximations for stochastic differential games. SIAM J Control Optim 40:457–486CrossRefMathSciNetGoogle Scholar
  18. Kushner HJ (2008) Numerical methods for controlled stochastic delay systems. Birkhäuser, BostonCrossRefMATHGoogle Scholar
  19. Kushner HJ, Dupuis PG (1992) Numerical methods for stochastic control problems in continuous time. Springer, New YorkCrossRefMATHGoogle Scholar
  20. Kushner HJ, Kleinman AJ (1968) Numerical methods for the solution of the degenerate nonlinear elliptic equations arising in optimal stochastic control theory. IEEE Trans Autom Control AC-13:344–353CrossRefMathSciNetGoogle Scholar
  21. Kushner HJ, Yin G (2003) Stochastic approximation and recursive algorithms and applications, 2nd edn. Springer, New YorkMATHGoogle Scholar
  22. Menaldi J (1989) Some estimates for finite difference approximations. SIAM J Control Optim 27:579–607CrossRefMATHMathSciNetGoogle Scholar
  23. Peng S (1990) A general stochastic maximum principle for optimal control problems. SIAM J Control Optim 28:966–979CrossRefMATHMathSciNetGoogle Scholar
  24. Sethi SP, Zhang Q (1994) Hierarchical decision making in stochastic manufacturing systems. Birkhäuser, BostonCrossRefMATHGoogle Scholar
  25. Song QS, Yin G (2009) Rates of convergence of numerical methods for controlled regime-switching diffusions with stopping times in the costs. SIAM J Control Optim 48:1831–1857CrossRefMATHMathSciNetGoogle Scholar
  26. Song QS, Yin G, Zhang Z (2006) Numerical method for controlled regime-switching diffusions and regime-switching jump diffusions. Automatica 42:1147–1157CrossRefMATHMathSciNetGoogle Scholar
  27. Song QS, Yin G, Zhang Z (2008) Numerical solutions for stochastic differential games with regime switching. IEEE Trans Autom Control 53:509–521CrossRefMathSciNetGoogle Scholar
  28. Warga J (1962) Relaxed variational problems. J Math Anal Appl 4:111–128CrossRefMATHMathSciNetGoogle Scholar
  29. Yan HM, Yin G, Lou SXC (1994) Using stochastic optimization to determine threshold values for control of unreliable manufacturing systems. J Optim Theory Appl 83:511–539CrossRefMATHMathSciNetGoogle Scholar
  30. Yin G, Zhang Q (2013) Continuous-time Markov chains and applications: a two-time-scale approach, 2nd edn. Springer, New YorkCrossRefGoogle Scholar
  31. Yin G, Zhu C (2010) Hybrid switching diffusions: properties and applications. Springer, New YorkCrossRefGoogle Scholar
  32. Yin G, Liu RH, Zhang Q (2002) Recursive algorithms for stock liquidation: a stochastic optimization approach. SIAM J Optim 13:240–263CrossRefMATHMathSciNetGoogle Scholar
  33. Yin G, Jin H, Jin Z (2009) Numerical methods for portfolio selection with bounded constraints. J Comput Appl Math 233:564–581CrossRefMATHMathSciNetGoogle Scholar
  34. Yong J, Zhou XY (1999) Stochastic controls: Hamiltonian systems and HJB equations. Springer, New YorkCrossRefMATHGoogle Scholar
  35. Zhang J (2006) Rate of convergence of finite difference approximations for degenerate ODEs. Math Comput 75:1755–1778CrossRefMATHGoogle Scholar

Copyright information

© Springer-Verlag London 2014

Authors and Affiliations

  1. 1.Department of Mathematics, Wayne State UniversityDetroit, MIUSA