Skip to main content
SpringerLink
Log in

HJB Equation, Dynamic Programming Principle, and Stochastic Optimal Control

  • Conference paper
  • First Online:
Nonlinear Partial Differential Equations for Future Applications (PDEFA 2017)

Part of the book series: Springer Proceedings in Mathematics & Statistics ((PROMS,volume 346))

  • 618 Accesses

Abstract

The paper is an extended version of lecture notes from a mini-course given by the author in the workshop Optimal Control and PDE in Tohoku University in 2017. The main objective of the lecture notes is to give a short but rigorous introduction to the dynamic programming approach to stochastic optimal control problems. The manuscript discusses, among other things, the classical necessary and sufficient conditions for optimality, properties of the value function, and it contains a proof of the dynamic programming principle, and a proof that the value function is a unique viscosity solution of the associated Hamilton-Jacobi-Bellman equation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 119.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 159.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 159.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bardi, M., Capuzzo-Dolcetta, I.: Optimal control and viscosity solutions of Hamilton-Jacobi-Bellman equations, with appendices by Maurizio Falcone and Pierpaolo Soravia. In: Systems & Control: Foundations & Applications. Birkhäuser Boston, Inc., Boston, MA (1997)

    Google Scholar 

  2. Bensoussan, A., Lions, J.-L.: Applications of variational inequalities in stochastic control, Translated from the French. In: Studies in Mathematics and its Applications, 12. North-Holland Publishing Co., Amsterdam-New York (1982)

    Google Scholar 

  3. Borkar, V.S.: Optimal Control of Diffusion Processes, Pitman Research Notes in Mathematics Series, vol. 203, Longman Scientific & Technical, Harlow; copublished in the United States with Wiley, Inc., New York (1989)

    Google Scholar 

  4. Bouchard, B., Touzi, N.: Weak dynamic programming principle for viscosity solutions. SIAM J. Control Optim. 49(3), 948–962 (2011)

    Article  MathSciNet  Google Scholar 

  5. Claisse, J., Talay, D., Tan, X.: A pseudo-Markov property for controlled diffusion processes. SIAM J. Control Optim. 54(2), 1017–1029 (2016)

    Article  MathSciNet  Google Scholar 

  6. Crandall, M.G., Ishii, H., Lions, P.L.: User’s guide to viscosity solutions of second order partial differential equations, Bull. Amer. Math. Soc. (N.S.) 27(1), 1–67 (1992)

    Google Scholar 

  7. El Karoui, N.: Les aspects probabilistes du contrôle stochastique. (French) [The probabilistic aspects of stochastic control] Ninth Saint Flour Probability Summer School-1979 (Saint Flour, 1979), pp. 73–238, Lecture Notes in Mathematics, 876, Springer, Berlin-New York (1981)

    Google Scholar 

  8. El Karoui, N., Nguyen, N.D., Jeanblanc-Picqué, M.: Compactification methods in the control of degenerate diffusions: existence of an optimal control. Stochastics 20(3), 169–219 (1987)

    Google Scholar 

  9. Fabbri, G., Gozzi, F., Święch, A.: Stochastic optimal control in infinite dimension, Dynamic programming and HJB equations, with a contribution by Marco Fuhrman and Gianmario Tessitore. In: Probability Theory and Stochastic Modelling, vol. 82. Springer, Cham (2017)

    Google Scholar 

  10. Fleming, W.H., Rishel, R.W.: Deterministic and stochastic optimal control. In: Applications of Mathematics, vol. 1. Springer, Berlin-New York (1975)

    Google Scholar 

  11. Fleming, W.H., Soner, H.M.: Controlled Markov processes and viscosity solutions. In: Stochastic Modelling and Applied Probability, vol. 25, 2nd edn. Springer, New York (2006)

    Google Scholar 

  12. Haussmann, U.G., Lepeltier, J.-P.: On the existence of optimal controls. SIAM J. Control Optim. 28(4), 851–902 (1990)

    Article  MathSciNet  Google Scholar 

  13. Krylov, N.V.: Controlled Diffusion Processes, Translated from the Russian by A. B. Aries. Applications of Mathematics, vol. 14. Springer, New York-Berlin (1980)

    Google Scholar 

  14. Lions, P.-L.: Optimal control of diffusion processes and Hamilton-Jacobi-Bellman equations, I. The dynamic programming principle and applications. Comm. Partial Differ. Equ. 8(10), 1101–1174 (1983)

    Google Scholar 

  15. Lions, P.-L.: Optimal control of diffusion processes and Hamilton-Jacobi-Bellman equations, II. Viscosity solutions and uniqueness. Comm. Partial Differ. Equ. 8(11), 1229–1276 (1983)

    Google Scholar 

  16. Morimoto, H.: Stochastic control and mathematical modeling. In: Applications in Economics, Encyclopedia of Mathematics and its Applications, vol. 131. Cambridge University Press, Cambridge (2010)

    Google Scholar 

  17. Nisio, M.: Some remarks on stochastic optimal controls. In: Proceedings of the Third Japan-USSR Symposium on Probability Theory (Tashkent, 1975). Lecture Notes in Mathematics, vol. 550, pp. 446–460. Springer, Berlin (1975)

    Google Scholar 

  18. Nisio, M.: Lectures on Stochastic Control Theory, ISI Lecture Notes, 9. Macmillan Co. of India Ltd, New Delhi (1981)

    MATH  Google Scholar 

  19. Nisio, M.: Stochastic control theory, dynamic programming principle. In: Probability Theory and Stochastic Modelling, vol. 72, 2nd edn. Springer, Tokyo (2015)

    Google Scholar 

  20. Ondreját, M.: Uniqueness for stochastic evolution equations in Banach spaces, Dissertationes Mathematicae (Rozprawy Mat.) 426 (2004), 63 pp

    Google Scholar 

  21. Pham, H.: Continuous-time stochastic control and optimization with financial applications. In: Stochastic Modelling and Applied Probability, vol. 61. Springer, Berlin (2009)

    Google Scholar 

  22. Soner, H.M., Touzi, N.: Dynamic programming for stochastic target problems and geometric flows. J. Eur. Math. Soc. (JEMS) 4(3), 201–236 (2002)

    Article  MathSciNet  Google Scholar 

  23. Touzi, N.: Optimal Stochastic Control, Stochastic Target Problems, and Backward SDE, with Chapter 13 by Angès Tourin, Fields Institute Monographs, vol. 29. Springer, New York; Fields Institute for Research in Mathematical Sciences, Toronto, ON (2013)

    Google Scholar 

  24. Yong, Y., Zhou, X.Y.: Stochastic controls, Hamiltonian systems and HJB equations. In: Applications of Mathematics (New York), vol. 43. Springer, New York (1999)

    Google Scholar 

  25. \(\check{\rm Z}\)itković, G.: Dynamic programming for controlled Markov families: abstractly and over martingale measures. SIAM J. Control Optim. 52(3), 1597–1621 (2014)

    Google Scholar 

Download references

Acknowledgements

I would like to thank Dr. Shota Tateyama for typing the first draft of the notes.

Author information

Authors and Affiliations

  1. School of Mathematics, Georgia Institute of Technology, Atlanta, GA, 30332, USA

    Andrzej Święch

Authors
  1. Andrzej Święch
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andrzej Święch .

Editor information

Editors and Affiliations

  1. Department of Applied Physics, Waseda University, Tokyo, Japan

    Shigeaki Koike

  2. Department of Mathematics, Waseda University, Tokyo, Japan

    Hideo Kozono

  3. Mathematical Institute, Tohoku University, Sendai, Japan

    Takayoshi Ogawa

  4. Graduate School of Information Sciences, Tohoku University, Sendai, Japan

    Shigeru Sakaguchi

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Święch, A. (2021). HJB Equation, Dynamic Programming Principle, and Stochastic Optimal Control. In: Koike, S., Kozono, H., Ogawa, T., Sakaguchi, S. (eds) Nonlinear Partial Differential Equations for Future Applications. PDEFA 2017. Springer Proceedings in Mathematics & Statistics, vol 346. Springer, Singapore. https://doi.org/10.1007/978-981-33-4822-6_5

Download citation

Publish with us

Policies and ethics

Search

Navigation