Skip to main content
SpringerLink
Log in

Approximate Solutions to the Time-Invariant Hamilton–Jacobi–Bellman Equation

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

In this paper, we develop a new method to approximate the solution to the Hamilton–Jacobi–Bellman (HJB) equation which arises in optimal control when the plant is modeled by nonlinear dynamics. The approximation is comprised of two steps. First, successive approximation is used to reduce the HJB equation to a sequence of linear partial differential equations. These equations are then approximated via the Galerkin spectral method. The resulting algorithm has several important advantages over previously reported methods. Namely, the resulting control is in feedback form and its associated region of attraction is well defined. In addition, all computations are performed off-line and the control can be made arbitrarily close to optimal. Accordingly, this paper presents a new tool for designing nonlinear control systems that adhere to a prescribed integral performance criterion.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Anderson, B. D. O., and Moore, J. B., Linear Optimal Control, Prentice-Hall, Englewood Cliffs, New Jersey, 1971.

    Google Scholar 

  2. Bryson, A. E., and Ho, Y. C., Applied Optimal Control, Hemisphere, New York, New York, 1975.

    Google Scholar 

  3. Kirk, D. E., Optimal Control Theory, Prentice-Hall, Englewood Cliffs, New Jersey, 1970.

    Google Scholar 

  4. Lewis, F. L., Optimal Control, John Wiley and Sons, New York, New York, 1986.

    Google Scholar 

  5. Sage, A. P., and White, C. C., III, Optimum Systems Control, 2nd Edition, Prentice-Hall, Englewood Cliffs, New Jersey, 1977.

    Google Scholar 

  6. Bosarge, W. E., Johnson, O. G., McKnight, R. S., and Timlake, W. P., The Ritz-Galerkin Procedure for Nonlinear Control Problems, SIAM Journal on Numerical Analysis, Vol. 10, pp. 94–110, 1973.

    Google Scholar 

  7. Hofer, E. P., and Tibken, B., An Iterative Method for the Finite-Time Bilinear-Quadratic Control Problem, Journal of Optimization Theory and Applications, Vol. 57, pp. 411–427, 1988.

    Google Scholar 

  8. Aganovic, Z., and Gajic, Z., The Successive Approximation Procedure for Finite-Time Optimal Control of Bilinear Systems, IEEE Transactions on Automatic Control, Vol. 39, pp. 1932–1935, 1994.

    Google Scholar 

  9. Cebuhar, W. A., and Costanza, V., Approximation Procedures for the Optimal Control of Bilinear and Nonlinear Systems, Journal of Optimization Theory and Applications, Vol. 43, pp. 615–627, 1984.

    Google Scholar 

  10. Rosen, O., and Luus, R., Global Optimization Approach to Nonlinear Optimal Control, Journal of Optimization Theory and Applications, Vol. 73, pp. 547–562, 1992.

    Google Scholar 

  11. Albrekht, E. G., On the Optimal Stabilization of Nonlinear Systems, Journal of Applied Mathematics and Mechanics, Vol. 25, pp. 836–844, 1961.

    Google Scholar 

  12. Lukes, D. L., Optimal Regulation of Nonlinear Dynamical Systems, SIAM Journal on Control and Optimization, Vol. 7, pp. 75–100, 1969.

    Google Scholar 

  13. Garrard, W. L., and Jordan, J. M., Design of Nonlinear Automatic Flight Control Systems, Automatica, Vol. 13, pp. 497–505, 1977.

    Google Scholar 

  14. Nishikawa, Y., Sannomiya, N., and Itakura, H., A Method for Suboptimal Design of Nonlinear Feedback Systems, Automatica, Vol. 7, pp. 703–712, 1971.

    Google Scholar 

  15. Werner, R. A., and Cruz, J. B., Feedback Control Which Preserves Optimality for Systems with Unknown Parameters, IEEE Transactions on Automatic Control, Vol. 13, pp. 621–629, 1968.

    Google Scholar 

  16. Halme, A., and Hamalainen, R. P., On the Nonlinear Regulator Problem, Journal of Optimization Theory and Applications, Vol. 16, pp. 255–275, 1975.

    Google Scholar 

  17. Ryan, E. P., Optimal Feedback Control of Bilinear Systems, Journal of Optimization Theory and Applications, Vol. 44, pp. 333–362, 1984.

    Google Scholar 

  18. Tzasfestas, S. G., Anagnostou, K. E., and Pimenides, T. G., Stabilizing Optimal Control of Bilinear Systems with a Generalized Cost, Optimal Control Applications and Methods, Vol. 5, pp. 111–117, 1984.

    Google Scholar 

  19. Lu, P., A New Nonlinear Optimal Feedback Control Law, Control Theory and Advanced Technology, Vol. 9, pp. 947–954, 1993.

    Google Scholar 

  20. Freeman, R. A., and Kokotovic, P. V., Optimal Nonlinear Controllers for Feedback Linearizable Systems, Proceedings of the American Control Conference, Seattle, Washington, pp. 2722–2726, 1995.

  21. Crandall, M. G., Ishii, H., and Lions, P. L., User's Guide to Viscosity Solutions of Second-Order Partial Differential Equations, Bulletin of the American Mathematical Society, Vol. 27, pp. 1–67, 1992.

    Google Scholar 

  22. Capuzzo Dolcetta, I., On a Discrete Approximation of the Hamilton-Jacobi Equation of Dynamic Programming, Applied Mathematics and Optimization, Vol. 10, pp. 367–377, 1983.

    Google Scholar 

  23. Cappuzzo Dolcetta, I., and Ishii, H., Approximate Solutions of the Bellman Equation of Deterministic Control Theory, Applied Mathematics and Optimization, Vol. 11, pp. 161–181, 1984.

    Google Scholar 

  24. Falcone, M., and Ferretti, R., Discrete Time High-Order Schemes for Viscosity Solutions of Hamilton-Jacobi-Bellman Equations, Numerische Mathematik, Vol. 67, pp. 315–344, 1994.

    Google Scholar 

  25. Capuzzo Dolcetta, I., and Falcone, M., Discrete Dynamic Programming and Viscosity Solutions of the Bellman Equation, Annales de l'Institut Henri Poincaré: Analyse Nonlineare, Vol. 6(Supplement), pp. 161–184, 1989.

    Google Scholar 

  26. Gonzalez, R., and Rofman, E., On Deterministic Control Problems: An Approximation Procedure for the Optimal Cost, I: The Stationary Problem, SIAM Journal on Control and Optimization, Vol. 23, pp. 242–266, 1985.

    Google Scholar 

  27. Gonzalez, R., and Rofman, E., On Deterministic Control Problems: An Approximation Procedure for the Optimal Cost, II: The Nonstationary Problem, SIAM Journal on Control and Optimization, Vol. 23, pp. 267–285, 1985.

    Google Scholar 

  28. Falcone, M., A Numerical Approach to the Infinite-Horizon Problem of Deterministic Control Theory, Applied Mathematics and Optimization, Vol. 15, pp. 1–13, 1987.

    Google Scholar 

  29. Kushner, H. J., Numerical Methods for Stochastic Control Problems in Continuous Time, SIAM Journal on Control and Optimization, Vol. 28, pp. 999–1048, 1990.

    Google Scholar 

  30. Fleming, W. H., and Soner, H. M., Controlled Markov Processes and Viscosity Solutions, Springer Verlag, Berlin, Germany, 1993.

    Google Scholar 

  31. Baumann, W. T., and Rugh, W. J., Feedback Control of Nonlinear Systems by Extended Linearization, IEEE Transactions on Automatic Control, Vol. 31, pp. 40–46, 1986.

    Google Scholar 

  32. Cloutier, J. R., D'Souza, C. N., and Mracek, C. P., Nonlinear Regulation and Nonlinear H ∞-Control Via the State-Dependent Riccati Equation Technique, IFAC World Congress, San Francisco, California, 1996.

  33. Goh, C. J., On the Nonlinear Optimal Regulator Problem, Automatica, Vol. 29, pp. 751–756, 1993.

    Google Scholar 

  34. Johansson, R., Quadratic Optimization of Motion Coordination and Control, IEEE Transactions on Automatic Control, Vol. 35, pp. 1197–1208, 1990.

    Google Scholar 

  35. Khalil, H. K., Nonlinear Systems, Macmillan Publishing Company, New York, New York, 1992.

    Google Scholar 

  36. Bellman, R. E., Dynamic Programming, Princeton University Press, Princeton, New Jersey, 1957.

    Google Scholar 

  37. Rekasius, Z. V., Suboptimal Design of Intentionally Nonlinear Controllers, IEEE Transactions on Automatic Control, Vol. 9, pp. 380–386, 1964.

    Google Scholar 

  38. Haussler, R. L., On the Suboptimal Design of Nonlinear Control Systems, PhD Thesis, Purdue University, Lafayette, Indiana, 1963.

    Google Scholar 

  39. Leake, R. J., and Liu, R. W., Construction of Suboptimal Control Sequences, SIAM Journal on Control and Optimization, Vol. 5, pp. 54–63, 1967.

    Google Scholar 

  40. Saridis, G. N., and Lee, C. S. G., An Approximation Theory of Optimal Control for Trainable Manipulators, IEEE Transactions on Systems, Man, and Cybernetics, Vol. 9, pp. 152–159, 1979.

    Google Scholar 

  41. Saridis, G. N., and Wang, F., Suboptimal Control of Nonlinear Stochastic Systems, Control Theory and Advanced Technology, Vol. 10, pp. 847–871, 1994.

    Google Scholar 

  42. Vaisbord, E. M., An Approximate Method for the Synthesis of Optimal Control, Automation and Remote Control, Vol. 24, pp. 1626–1632, 1963.

    Google Scholar 

  43. Milshtein, G. N., Successive Approximations for Solution of One Optimal Problem, Automation and Remote Control, Vol. 25, pp. 298–306, 1964.

    Google Scholar 

  44. Saridis, G. N., and Balaram, J., Suboptimal Control for Nonlinear Systems, Control Theory and Advanced Technology, Vol. 2, pp. 547–462, 1986.

    Google Scholar 

  45. Bertsekas, D. P., On Error Bounds for Successive Approximation Methods, IEEE Transactions on Automatic Control, Vol. 21, pp. 394–396, 1976.

    Google Scholar 

  46. Kleinman, D. L., On an Iterative Technique for Riccati Equation Computations, IEEE Transactions on Automatic Control, Vol. 13, pp. 114–115, 1968.

    Google Scholar 

  47. Kleinman, D. L., An Easy Way to Stabilize a Linear Constant System, IEEE Transactions on Automatic Control, Vol. 15, 1970.

  48. Sandell, N. R., On Newton's Method for Riccati Equation Solution, IEEE Transactions on Automatic Control, Vol. 19, pp. 254–255, 1974.

    Google Scholar 

  49. Mageirou, E. F., Iterative Techniques for Riccati Game Equations, Journal of Optimization Theory and Applications, Vol. 22, pp. 51–61, 1977.

    Google Scholar 

  50. Laub, A. J., Invariant Subspace Methods for the Numerical Solution of Riccati Equations, The Riccati Equation, Edited by S. Bittanti and W. Laub, Springer Verlag, New York, New York, pp. 163–196, 1991.

    Google Scholar 

  51. Balaram, J., Suboptimal Control of Nonlinear Systems, PhD Thesis, Rensselaer Polytechnic Institute, Troy, New York, 1985.

    Google Scholar 

  52. Beard, R., Improving the Closed-Loop Performance of Nonlinear Systems, PhD Thesis, Rensselaer Polytechnic Institute, Troy, New York, 1995.

    Google Scholar 

  53. Glad, T., Robust Nonlinear Regulators Based on Hamilton-Jacobi Theory and Lyapunov Functions, IEEE Control Conference, Cambridge, Massachusetts, pp. 276–280, 1985.

  54. Glad, S. T., Robustness of Nonlinear State Feedback: A Survey, Automatica, Vol. 23, pp. 425–435, 1987.

    Google Scholar 

  55. Tsitsiklis, J. N., and Athans, M., Guaranteed Robustness Properties of Multivariable Nonlinear Stochastic Optimal Regulators, IEEE Transactions on Automatic Control, Vol. 29, pp. 690–696, 1984.

    Google Scholar 

  56. Kantorovich, L. V., and Krylov, V. I., Approximate Methods of Higher Analysis, Interscience Publishers, New York, New York, 1958.

    Google Scholar 

  57. Mikhlin, S. G., Variational Methods in Mathematical Physics, Macmillan Company, New York, New York, 1964.

    Google Scholar 

  58. Mikhlin, S. G., and Smolitskiy, K. L., Approximate Methods for Solution of Differential and Integral Equations, American Elsevier Publishing Company, New York, New York, 1967.

    Google Scholar 

  59. Petryshyn, W. V., On a Class of K-p.d. and Non-K-p.d. Operators and Operator Equations, Journal of Mathematical Analysis and Applications, Vol. 10, pp. 1–24, 1965.

    Google Scholar 

  60. Finlayson, B. A., The Method of Weighted Residuals and Variational Principles, Academic Press, New York, New York, 1972.

    Google Scholar 

  61. Zeidler, E., Nonlinear Functional Analysis and Its Applications, II/A: Linear Monotone Operators, Springer Verlag, Berlin, Germany, 1990.

    Google Scholar 

  62. Zeidler, E., Nonlinear Functional Analysis and Its Applications, II/B: Nonlinear Monotone Operators, Springer Verlag, Berlin, Germany, 1990.

    Google Scholar 

  63. Bittanti, S., Laub, A., and Willems, J. C., The Riccati Equation, Springer Verlag, New York, New York, 1991.

    Google Scholar 

  64. Beard, R. W., Saridis, G. N., and Wen, J. T., Galerkin Approximation of the Generalized Hamilton-Jacobi-Bellman Equation, Automatica, Vol. 33, 1997.

  65. Apostol, T. M., Mathematical Analysis, Addison-Wesley, Reading, Massachusetts, 1974.

    Google Scholar 

  66. Stevenson, W. D., Elements of Power System Analysis, McGraw-Hill, New York, New York, 1982.

    Google Scholar 

  67. Wang, Y., Hill, D. J., Middleton, R. H., and Gao, L., Transient Stability Enhancement and Voltage Regulation of Power Systems, IEEE Transactions on Power Systems, Vol. 8, pp. 620–627, 1993.

    Google Scholar 

  68. King, C. A., Chapman, J. W., and Llic, M. D., Feedback Linearizing Excitation Control on a Full-Scale Power System Model, IEEE Transactions on Power Systems, Vol. 9, pp. 1102–1109, 1994.

    Google Scholar 

  69. Gao, L., Chen, L., Fan, Y., and Ma, H., A Nonlinear Control Design for Power Systems, Automatica, Vol. 28, pp. 975–979, 1992.

    Google Scholar 

  70. Marino, R., An Example of a Nonlinear Regulator, IEEE Transactions on Automatic Control, Vol. 29, pp. 276–279, 1984.

    Google Scholar 

  71. Wang, Y., Hill, D. J., Middleton, R. H., and Gao, L., Transient Stabilization of Power Systems with an Adaptive Control Law, Automatica, Vol. 30, pp. 1409–1413, 1994.

    Google Scholar 

  72. Chapman, J. W., Ilic, M. D., King, C. A., Eng, L., and Kaufman, H., Stabilizing a Multimachine Power System via Decentralized Feedback Linearizing Excitation Control, IEEE Transactions on Power Systems, Vol. 8, pp. 830–839, 1993.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Brigham Young University, Provo, Utah

    R. W. Beard (Assistant Professor)

  2. Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute, Troy, New York

    G. N. Saridis (Professor) & J. T. Wen

Authors
  1. R. W. Beard
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. G. N. Saridis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. J. T. Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Beard, R.W., Saridis, G.N. & Wen, J.T. Approximate Solutions to the Time-Invariant Hamilton–Jacobi–Bellman Equation. Journal of Optimization Theory and Applications 96, 589–626 (1998). https://doi.org/10.1023/A:1022664528457

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1022664528457

Search

Navigation