Skip to main content
SpringerLink
Log in

Error estimation and adaptive discretization for the discrete stochastic Hamilton–Jacobi–Bellman equation

  • Published:
Numerische Mathematik Aims and scope Submit manuscript

Summary.

Generalizing an idea from deterministic optimal control, we construct a posteriori error estimates for the spatial discretization error of the stochastic dynamic programming method based on a discrete Hamilton–Jacobi–Bellman equation. These error estimates are shown to be efficient and reliable, furthermore, a priori bounds on the estimates depending on the regularity of the approximate solution are derived. Based on these error estimates we propose an adaptive space discretization scheme whose performance is illustrated by two numerical examples.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Bardi, M., Dolcetta, I.C.: Optimal Control and Viscosity Solutions of Hamilton-Jacobi-Bellman equations. Birkhäuser, Boston, 1997

  2. Barles, G., Jakobsen, E.R.: On the convergence rate of approximation schemes for Hamilton–Jacobi–Bellman equations. M2AN, Math. Model. Numer. Anal. 36, 33–54 (2002)

    Google Scholar 

  3. Bertsekas, D.P.: Dynamic Programming and Optimal Control. Vol. 1 and 2. Athena Scientific, Belmont, MA, 1995

  4. Camilli, F., Falcone, M.: An approximation scheme for the optimal control of diffusion processes. RAIRO, Modélisation Math. Anal. Numér. 29, 97–122 (1995)

    Google Scholar 

  5. Camilli, F., Grüne, L.: Numerical approximation of the maximal solutions for a class of degenerate Hamilton–Jacobi equations. SIAM J. Numer. Anal. 38, 1540–1560 (2000)

    Article  MATH  Google Scholar 

  6. Camilli, F., Grüne, L.: Characterizing attraction probabilities via the stochastic Zubov equation. Discrete Contin. Dyn. Syst. Ser. B 3, 457–468 (2003)

    Google Scholar 

  7. Camilli, F., Grüne, L., Wirth, F.: A regularization of Zubov’s equation for robust domains of attraction. In: Nonlinear Control in the Year 2000, Volume 1, A. Isidori, F. Lamnabhi-Lagarrigue, and W. Respondek, eds. Lecture Notes in Control and Information Sciences 258, NCN, Springer Verlag, London, 2000, pp. 277–290

  8. Camilli, F., Grüne, L., Wirth, F.: Characterizing controllability probabilities of stochastic control systems via Zubov’s method. In: Proceedings of the 16th International Symposium on Mathematical Theory of Networks and Systems, Leuven, Belgium, 2004, CD-ROM, Paper No. 65

  9. Camilli, F., Loreti, P.: Zubov’s method for stochastic differential equations. NoDEA Nonlinear Differ. Equ. Appl., (2004), To appear

  10. Carlini, E., Falcone, M., Ferretti, R.: An efficient algorithm for Hamilton–Jacobi equations in high dimensions. Comput. Vis. Sci. (2004), volume 7, pp. 15–29

  11. Colonius, F., Kliemann, W.: The Dynamics of Control. Birkhäuser, Boston, 2000

  12. Crandall, M.G., Lions, P.L.: Viscosity solutions of Hamilton–Jacobi equations. Trans. Amer. Math. Soc. 277, 1–42 (1983)

    MathSciNet  MATH  Google Scholar 

  13. Daniel, J.W.: Splines and efficiency in dynamic programming. J. Math. Anal. Appl. 54, 402–407 (1976)

    Article  MATH  Google Scholar 

  14. Falcone, M.: A numerical approach to the infinite horizon problem of deterministic control theory. Appl. Math. Optim. 15, 1–13 (1987); Corrigenda. ibid. 23, 213–214 (1991)

    MATH  Google Scholar 

  15. Falcone, M., Ferretti, R., Manfroni, T.: Optimal discretization steps in semi-Lagrangian approximation of first-order PDEs. In: Numerical methods for viscosity solutions and applications, M. Falcone, C. Makridakis (eds.), vol. 59 of Ser. Adv. Math. Appl. Sci. World Scientific, Singapore, 2001, pp. 95–117

  16. Ferretti, R.: Convergence of semi-Lagrangian approximations to convex Hamilton-Jacobi equations under (very) large Courant numbers. SIAM J. Numer. Anal. 40, 2240–2253 (2003)

    Article  MATH  Google Scholar 

  17. Fleming, W.H., Soner, M.H.: Controlled Markov processes and viscosity solutions. Springer–Verlag, New York, 1993

  18. Franke, R., Terwiesch, P., Meyer, M.: Development of an algorithm for the optimal control of trains. In: Proceedings of the 39th IEEE Conference on Decision and Control, Sydney, Australia, 2000, pp. 2123–2128

  19. González, R.L.V., Sagastizábal, C.A.: Un algorithme pour la résolution rapide d’équations discrètes de Hamilton-Jacobi-Bellman. C. R. Acad. Sci. Paris, Sér. I 311, 45–50 (1990)

    Google Scholar 

  20. Grüne, L.: An adaptive grid scheme for the discrete Hamilton–Jacobi–Bellman equation. Numer. Math. 75, 319–337 (1997)

    Article  Google Scholar 

  21. Grüne, L.: Homogeneous state feedback stabilization of homogeneous systems. SIAM J. Control Optim. 38, 1288–1314 (2000)

    Article  Google Scholar 

  22. Grüne, L.: Adaptive grid generation for evolutive Hamilton–Jacobi–Bellman equations. In: Numerical methods for viscosity solutions and applications, M. Falcone and C. Makridakis, eds., vol. 59 of Ser. Adv. Math. Appl. Sci. World Scientific, Singapore, 2001, pp. 153–172

  23. Grüne, L.: Asymptotic Behavior of Dynamical and Control Systems under Perturbation and Discretization. Lecture Notes in Mathematics, Vol. 1783, Springer–Verlag, 2002

  24. Grüne, L., Metscher, M., Ohlberger, M.: On numerical algorithm and interactive visualization for optimal control problems. Comput. Vis. Sci. 1, 221–229 (1999)

    Article  Google Scholar 

  25. Grüne, L., Semmler, W.: Using dynamic programming with adaptive grid scheme for optimal control problems in economics. J. Econ. Dyn. Control (2004), To appear

  26. Grüne, L., Semmler, W., Sieveking, M.: Creditworthyness and thresholds in a credit market model with multiple equilibria. Econ. Theory, (2005), pp. 287–315

  27. Kloeden, P.E., Platen, E.: Numerical Solution of Stochastic Differential Equations. Springer–Verlag, Heidelberg, 1992. (3rd revised and updated printing, 1999)

  28. Kushner, H.J., Dupuis, P.G.: Numerical Methods for Stochastic Control Problems in Continuous Time. Applications of Mathematics, 24, Springer-Verlag, New York, 2nd ed. 2001

  29. Lions, P.L.: Generalized solutions of Hamilton-Jacobi equations. Pitman, London, 1982

  30. Menaldi, J.L.: Some estimates for finite difference approximations. SIAM J. Control Optim. 27, 579–607 (1989)

    MATH  Google Scholar 

  31. Munos, R., Moore, A.: Variable resolution discretization in optimal control. Mach. Learn. (2002), pp. 291–323

  32. Reiter, M.: Solving higher–dimensional continuous–time stochastic control problems by value function regression. J. Econ. Dyn. Control 23, 1329–1353 (1999)

    Article  MATH  Google Scholar 

  33. Rust, J.: Numerical dynamic programming in economics. In: Handbook of Computational Economics, H.M. Amman, D.A. Kendrick, and J. Rust, eds. Elsevier, Amsterdam, 1996

  34. Rust, J.: Using randomization to break the curse of dimensionality. Econometrica, 65, 487–516 (1997)

    Google Scholar 

  35. Sagona, M., Seghini, A.: An adaptive scheme on unstructured grids for the shape–from–shading problem. In: Numerical methods for viscosity solutions and applications. M. Falcone, C. Makridakis (eds.), vol. 59 of Ser. Adv. Math. Appl. Sci., World Scientific, Singapore, 2001

  36. Santos, M.S., Vigo-Aguiar, J.: Accuracy estimates for a numerical approach to stochastic growth models. Discussion Paper 107, Institute for Empirical Macroeconomics, Federal Reserve Bank of Minneapolis, December 1995

  37. Santos, M.S., Vigo-Aguiar, J.: Analysis of a numerical dynamic programming algorithm applied to economic models. Econometrica 66, 409–426 (1998)

    MATH  Google Scholar 

  38. Seeck, A.: Iterative Lösungen der Hamilton-Jacobi-Bellman-Gleichung bei unendlichem Zeithorizont. Diplomarbeit, Universität Kiel, 1997

  39. Siebert, K.G.: An a posteriori error estimator for anisotropic refinement. Numer. Math. 73, 373–398 (1996)

    Article  MATH  Google Scholar 

  40. Trick, M.A., Zin, S.E.: Spline approximations to value functions: a linear programming approach. Macroecon. Dyn. 1, 255–277 (1997)

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Mathematisches Institut, Universität Bayreuth, 95440, Bayreuth, Germany

    Lars Grüne

Authors
  1. Lars Grüne
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lars Grüne.

Additional information

Mathematics Subject Classification (2000): 93E20, 65N50, 49L20, 49M25, 65N15

Acknowledgments. This research was supported by the Center for Empirical Macroeconomics, University of Bielefeld. The support is gratefully acknowledged. I would also like to thank an anonymous referee who suggested several improvements for the paper.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Grüne, L. Error estimation and adaptive discretization for the discrete stochastic Hamilton–Jacobi–Bellman equation. Numer. Math. 99, 85–112 (2004). https://doi.org/10.1007/s00211-004-0555-4

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00211-004-0555-4

Keywords

Search

Navigation