Skip to main content
SpringerLink
Log in

Temporal Difference Methods for the Maximal Solution of Discrete-Time Coupled Algebraic Riccati Equations

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

In this paper, we present an iterative technique for deriving the maximal solution of a set of discrete-time coupled algebraic Riccati equations, based on temporal difference methods, which are related to the optimal control of Markovian jump linear systems and have been studied extensively over the last few years. We trace a parallel with the theory of temporal difference algorithms for Markovian decision processes to develop a λ-policy iteration like algorithm for the maximal solution of these equations. For the special cases in which λ=0 and λ=1, we have the situation in which the algorithm reduces to the iterations of the Riccati difference equations (value iteration) and quasilinearization method (policy iteration), respectively. The advantage of the proposed method is that an appropriate choice of λ between 0 and 1 can speed up the convergence of the policy evaluation step of the policy iteration method by using value iteration.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Mariton, M., Jump Linear Systems in Automatic Control, Marcel Dekker, New York, NY, 1990.

    Google Scholar 

  2. Costa, O. L. V., and Fragoso, M. D., Discrete-Time LQ-Optimal Control Problems for Infinite Marko Jump Parameter Systems, IEEE Transactions on Automatic Control, Vol. 40, pp. 2076–2088, 1995.

    Google Scholar 

  3. Ji, Y., and Chizeck, H. J., Controllability, Observability, and Discrete-Time Markovian Jump Linear Quadratic Control, International Journal of Control, Vol. 48, pp. 481–498, 1988.

    Google Scholar 

  4. Ji, Y., Chizeck, H. J., Feng, X., and Loparo, K. A., Stability and Control of Discrete-Time Jump Linear Systems, Control Theory and Advanced Technology, Vol. 7, pp. 247–270, 1991.

    Google Scholar 

  5. Abou-Kandil, H., Freiling, G., and Jank, G., On the Solution of Discrete-Time Markovian Jump Linear-Quadratic Control Problems, Automatica, Vol. 31, pp. 765–768, 1995.

    Google Scholar 

  6. Rami, M. A., and El Ghaoui, L., LMI Optimization for Nonstandard Riccati Equations Arising in Stochastic Control, IEEE Transactions on Automatic Control, Vol. 41, pp. 1666–1671, 1996.

    Google Scholar 

  7. Costa, O. L. V., Do Val, J. B. R., and Geromel, J. C., A Convex Programming Approach to ℋ2 -Control of Discrete-Time Markovian Jump Linear Systems, International Journal of Control, Vol. 66, pp. 557–579, 1997.

    Google Scholar 

  8. Do Val, J. B. R., Geromel, J. C., and Costa, O. L. V., Uncoupled Riccati Iterations for the Linear-Quadratic Control Problem of Discrete-Time Markov Jump Linear Systems, IEEE Transactions on Automatic Control, Vol. 43, pp. 1727–1733, 1998.

    Google Scholar 

  9. Do Val, J. B. R., Geromel, J. C., and Costa, O. L. V., Solution for the Linear-Quadratic Control Problem of Marko Jump Linear Systems, Journal of Optimization Theory and Applications, Vol. 103, pp. 283–311, 1999.

    Google Scholar 

  10. Gajic, Z., and Borno, I., Lyapuno Iterations for Optimal Control of Jump Linear Systems at Steady State, IEEE Transactions on Automatic Control, Vol. 40, pp. 481–498, 1995.

    Google Scholar 

  11. Costa, O. L. V., and Boukas, E. K., Necessary and Sufficient Condition for Robust Stability of Continuous-Time Linear Systems with Markovian Jumps, Journal of Optimization Theory and Applications, Vol. 99, pp. 359–379, 1998.

    Google Scholar 

  12. Bertsekas, D. P., and Tsitsilklis, J.N., Neurodynamic Programming, Athena Scientific, Belmont, Massachusetts, 1996.

    Google Scholar 

  13. Sutton, R. S., and Barto, A. G., Reinforcement Learning: An Introduction, MIT Press, Cambridge, Massachusetts, 1998.

    Google Scholar 

  14. Costa, O. L. V., and Fragoso, M. D., Stability Results for Discrete-Time Linear Systems with Markovian Jumping Parameters, Journal of Mathematical Analysis and Applications, Vol. 179, pp. 154–178, 1993.

    Google Scholar 

  15. Mariton, M., Almost Sure and Moment Stability of Jump Linear Systems, Systems and Control Letters, Vol. 11 pp. 393–397, 1988.

    Google Scholar 

  16. Costa, O. L. V., and Marques, R. P., Maximal and Stabilizing Hermitian Solutions for Discrete-Time Coupled Algebraic Riccati Equations, Mathematics of Control Signals and Systems, Vol. 12, pp. 167–195, 1999.

    Google Scholar 

  17. Blair, W. P., Jr., and Sworder, D. D., Feedback Control of a Class of Linear Discrete System with Jump Parameters and Quadratic Cost Criteria, International Journal of Control, Vol. 21, pp. 833–841, 1975.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Departamento de Engenharia de Telecomunicações e Controle, Escola Politécnica da Universidade de São Paulo, São Paulo, SP, Brazil

    O. L. V. COSTA (Professor)

  2. Departamento de Engenharia de Telecomunicações e Controle, Escola Politécnica da Universidade de São Paulo, São Paulo, SP, Brazil

    J. C. C. AYA (PhD Student)

Authors
  1. O. L. V. COSTA
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. J. C. C. AYA
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

COSTA, O.L.V., AYA, J.C.C. Temporal Difference Methods for the Maximal Solution of Discrete-Time Coupled Algebraic Riccati Equations. Journal of Optimization Theory and Applications 109, 289–309 (2001). https://doi.org/10.1023/A:1017510321237

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1017510321237

Search

Navigation