Skip to main content
SpringerLink
Log in

Numerical convergence for the Bellman equation of stochastic optimal control with quadratic costs and constraints

  • Published:
Dynamics and Control

Abstract

A predictor-corrector, Crank-Nicolson computer algorithm is examined for the Bellman equation of stochastic optimal control with quadratic costs and constrained control. A linearized comparison equation is heuristically derived for the nonlinear and discontinuous Bellman equation. Convergence of the method is studied using von Neumann's Fourier stability method. A mesh-ratio-type condition for the convergence is derived for the comparison equation. This condition is uniform for both parabolic and hyperbolic versions of the nonlinear equation. The results are valid for Gaussian stochastic noise and Poisson noise.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. W.H. Fleming, chairman,Future Directions in Control Theory: A Mathematical Prospective. Society for Industrial and Applied Mathematics: Philadelphia, PA, 1988.

    Google Scholar 

  2. D. Ludwig, “Optimal harvesting of a randomly fluctuating resource I: Application of perturbation methods,”SIAM J. Appl. Math., vol. 37, pp. 166–184, August 1979.

    Article  Google Scholar 

  3. D. Ludwig and J.M. Varah, “Optimal harvesting of a randomly fluctuating resource II: Numerical methods and results,”SIAM J. Appl. Math., vol. 37, pp. 185–205, August 1979.

    Article  Google Scholar 

  4. D. Ryan and F.B. Hanson, “Optimal harvesting of a logistic population in an environment with stochastic jumps,”J. Math. Biol., vol. 24, pp. 259–277, 1986.

    PubMed  Google Scholar 

  5. F.B. Hanson, “Bioeconomic model of the Lake Michigan alewife fishery,”Can. J. Fish. Aquat. Sci., vol. 44 (suppl. II), pp. 298–305, 1987.

    Google Scholar 

  6. F.B. Hanson and D. Ryan, “Optimal harvesting in a stochastic environment with density dependent jumps,” inMathematical Models of Renewable Resources, vol. 3, edited by R. Lamberson. Association of Resource Modelers: Arcata, CA, pp. 117–123, 1984.

    Google Scholar 

  7. F.B. Hanson and D. Ryan, “Optimal harvesting with density dependent random effects,”Natural Res. Modeling, vol. 2, pp. 439–455, Winter 1987.

    Google Scholar 

  8. M. Athans, D. Castanon, K.P. Dunn, C.S. Greene, W.H. Lee, N.R. Sandell, Jr., and A.S. Willsky, “The stochastic control of the F-8C aircraft using a multiple model adaptive control (MMAC) method—Part I: Equilibrium flight,”IEEE Trans. Autom. Control, vol. AC-22, pp. 768–780, October 1977.

    Article  Google Scholar 

  9. A.E. Bryson and Y.C. Ho,Applied Optimal Control. Hemisphere: New York, 1975.

    Google Scholar 

  10. R.E. Larson,State Increment Dynamic Programming. American Elsevier: New York, 1968.

    Google Scholar 

  11. D.H. Jacobson and D.Q. Mayne,Differential Dynamic Programming. American Elsevier: New York, 1970.

    Google Scholar 

  12. P. Dyer and S.R. McReynolds,The Computation and Theory of Optimal Control. Academic Press: New York, 1970.

    Google Scholar 

  13. I.H. Mufti,Computational Methods in Optimal Control Problems. Lecture Notes in Operations Research and Mathematical Systems, vol. 27. Springer-Verlag: Berlin, 1970.

    Google Scholar 

  14. E. Polak,Computational Methods in Optimization. Academic Press: New York, 1971.

    Google Scholar 

  15. E. Polak, “An historical survey of computational methods in optimal control,”SIAM Rev., vol. 15, pp. 553–584, 1973.

    Article  Google Scholar 

  16. H.J. Kushner, “A survey of some applications of probability and stochastic control theory to finite difference methods for degenerate elliptic and parabolic equations,”SIAM Rev., vol. 18, pp. 545–577, 1976.

    Article  Google Scholar 

  17. H.J. Kushner,Probability Methods for Approximations in Stochastic Control and for Elliptic Equations. Academic Press: New York, 1978.

    Google Scholar 

  18. H.J. Kusher and P.G. Dupuis,Numerical Methods for Stochastic Control in Continuous Time. Springer-Verlag: New York, 1992.

    Google Scholar 

  19. M.G. Crandall and P.-L. Lions, “Viscosity solutions of Hamilton-Jacobi equations,”Trans. Am. Math. Soc., vol. 277, pp. 1–42, 1983.

    Google Scholar 

  20. P.E. Souganidis, “Approximation schemes for viscosity solutions of Hamilton-Jacobi equations,”J. Diff. Eqns., vol. 59, pp. 1–43, 1985.

    Article  Google Scholar 

  21. R. Jensen, “The maximum principle for viscosity solutions of fully nonlinear second order partial differential equations,”Arch. Rat. Mech. Anal., vol. 101, pp. 1–27, 1988.

    Article  Google Scholar 

  22. M.G. Crandall, H. Ishii, and P.-L. Lions, “User's guide to viscosity solutions of second order partial differential equations,”Bull. Am. Math. Soc., vol. 27(1), pp. 1–67, July 1992.

    Google Scholar 

  23. P.-L. Lions and B. Perthame, “Remarks on Hamilton-Jacobi-Equations with measurable time-dependent Hamiltonians,”Nonlin. Anal. Th. Meth. Appl., vol. 11, pp. 613–622, 1987.

    Article  Google Scholar 

  24. I.I. Gihman and A.V. Skorohod,Stochastic Differential Equations. Springer-Verlag: New York, 1972.

    Google Scholar 

  25. I.I. Gihman and A.V. Skorohod,Controlled Stochastic Processes. Springer-Verlag: New York, 1979.

    Google Scholar 

  26. R.D. Richtmyer and K.W. Morton,Difference Methods for Initial-Value Problems. Wiley: New York, 1967.

    Google Scholar 

  27. A.R. Mitchell and D.F. Griffiths,The Finite Difference Method in Partial Differential Equations. Wiley: New York, 1980.

    Google Scholar 

  28. L. Arnold,Stochastic Differential Equations: Theory and Applications. Wiley: New York, 1974.

    Google Scholar 

  29. W.H. Fleming and R.W. Rishel,Deterministic and Stochastic Optimal Control. Springer-Verlag: New York, 1975.

    Google Scholar 

  30. Z. Schuss,Theory and Applications of Stochastic Differential Equations. Wiley: New York, 1980.

    Google Scholar 

  31. R.E. Bellman,Adaptive Control Processes: A Guided Tour. Princeton University Press: Princeton, NJ, 1961.

    Google Scholar 

  32. C.C. Holt, F. Modigliani, J. Muth, and H. Simon,Planning Production Inventories, and Work Force. Prentice-Hall: Englewood Cliffs, NJ, 1960.

    Google Scholar 

  33. M. Athans, “The role and use of the stochastic linear-quadratic-Gaussian problem in control system design,”IEEE Trans. Autom. Control, vol. AC-16, pp. 529–552, 1971.

    Article  Google Scholar 

  34. D.J. Bell and D.H. Jacobson,Singular Optimal Control. Academic Press: New York, 1975.

    Google Scholar 

  35. A. Jameson and R.E. O'Malley, Jr., “Cheap control of time-invariant regulator,”Appl. Math. Optim., vol. 1(4), pp. 337–354, 1975.

    Article  Google Scholar 

  36. R.E. O'Malley, Jr., and A. Jameson, “Singular perturbation and singular arcs—Part I,”IEEE Trans. Autom. Control, vol. AC-20, pp. 218–226, 1975.

    Article  Google Scholar 

  37. P. Kokotovic, H.K. Khalil, and J. O'Reilly,Singular Perturbation Methods in Control: Analysis and Design. Academic Press: New York, 1986.

    Google Scholar 

  38. J.E. Flaherty and R.E. O'Malley, Jr., “On computation of singular controls,”IEEE Trans. Autom. Control, vol. AC-22, pp. 640–648, 1977.

    Article  Google Scholar 

  39. J. Douglas, Jr., and T. DuPont, “Galerkin methods for parabolic equations,”SIAM J. Num. Anal., vol. 7, pp. 575–626, 1970.

    Article  Google Scholar 

  40. J. Douglas, Jr., “Effective time-stepping methods for the numerical solution of nonlinear parabolic problems,” inThe Mathematics of Finite Elements and Applications V: MAFELAP 1978, edited by J.R. Whiteman. Academic Press: New York, pp. 289–304, 1979.

    Google Scholar 

  41. F.B. Hanson, “Computational dynamic programming on a vector multiprocessor,”IEEE Trans. Autom. Control, vol. 36(4), pp. 507–511, 1991.

    Article  MathSciNet  Google Scholar 

  42. S.-L. Chung, F.B. Hanson, and H.H. Xu, “Parallel stochastic dynamic programming: Finite element methods,”Lin. Alg. Appl., vol. 172, pp. 197–218, July 1992.

    Article  Google Scholar 

  43. S.-L. Chung, F.B. Hanson, and H. Xu, “Supercomputer optimizations for stochastic optimal control applications,”Proc. 4th NASA Workshop on Computational Control of Flexible Aerospace Systems, NASA Conf. Publ. 10065, Part 1, edited by L.W. Taylor, NASA Langley Research Center, pp. 57–70, March 1991.

  44. H.H. Xu, F.B. Hanson, and S.-L. Chung, “Optimal data parallel methods for stochastic dynamic programming,”Proc. 1991 Int. Conf. Parallel Processing, vol. III Algorithms and Applications, CRC Press: Boca Raton, FL, pp. 142–146, August 1991.

  45. W. Feller,An Introduction to Probability Theory and Its Applications, vol. 2, Wiley, New York, 1971.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Northeastern Illinois University, 5500 North St. Louis Avenue, 60625-4699, Chicago, IL

    K. Naimipour

  2. Laboratory for Advanced Computing, Department of Mathematics, Statistics, and Computer Science, University of Illinois at Chicago, 851 S. Morgan St., M/C 249, 60607-7045, Chicago, IL

    F. B. Hanson

Authors
  1. K. Naimipour
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. F. B. Hanson
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Naimipour, K., Hanson, F.B. Numerical convergence for the Bellman equation of stochastic optimal control with quadratic costs and constraints. Dynamics and Control 3, 237–259 (1993). https://doi.org/10.1007/BF01972698

Download citation

  • Received:

  • Revised:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF01972698

Keywords

Search

Navigation