Journal of Zhejiang University SCIENCE C

, Volume 15, Issue 1, pp 43–50 | Cite as

Adaptive dynamic programming for linear impulse systems

  • Xiao-hua Wang
  • Juan-juan Yu
  • Yao Huang
  • Hua Wang
  • Zhong-hua Miao
Article

Abstract

We investigate the optimization of linear impulse systems with the reinforcement learning based adaptive dynamic programming (ADP) method. For linear impulse systems, the optimal objective function is shown to be a quadric form of the pre-impulse states. The ADP method provides solutions that iteratively converge to the optimal objective function. If an initial guess of the pre-impulse objective function is selected as a quadratic form of the pre-impulse states, the objective function iteratively converges to the optimal one through ADP. Though direct use of the quadratic objective function of the states within the ADP method is theoretically possible, the numerical singularity problem may occur due to the matrix inversion therein when the system dimensionality increases. A neural network based ADP method can circumvent this problem. A neural network with polynomial activation functions is selected to approximate the pre-impulse objective function and trained iteratively using the ADP method to achieve optimal control. After a successful training, optimal impulse control can be derived. Simulations are presented for illustrative purposes.

Key words

Adaptive dynamic programming (ADP) Impulse system Optimal control Neural network 

CLC number

TP273.1 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Ahmed, N.U., 2003. Existence of optimal controls for a general class of impulsive systems on Banach spaces. SIAM J. Control Optim., 42(2):669–685. [doi:10.1137/S0363012901391299]CrossRefMATHMathSciNetGoogle Scholar
  2. Bainov, D.D., Simeonov, P.S., 1995. Impulsive Differential Equations: Asymptotic Properties of the Solutions. World Scientific, Singapore.MATHGoogle Scholar
  3. Balakrishnan, S.N., Ding, J., Lewis, F.L., 2008. Issues on stability of ADP feedback controllers for dynamical systems. IEEE Tran. Syst. Man Cybern. B, 38(4):913–917. [doi:10.1109/TSMCB.2008.926599]CrossRefGoogle Scholar
  4. Bertsekas, D.P., 2011. Approximate policy iteration: a survey and some new methods. J. Control Theory Appl., 9(3):310–335.CrossRefMATHMathSciNetGoogle Scholar
  5. Dierks, T., Jagannathan, S., 2011. Online optimal control of nonlinear discrete-time systems using approximate dynamic programming. J. Control Theory Appl., 9(3):361–369. [doi:10.1007/s11768-011-0178-0]CrossRefMathSciNetGoogle Scholar
  6. Fraga, S.L., Pereira, F.L., 2012. Hamilton-Jacobi-Bellman equation and feedback synthesis for impulsive control. IEEE Trans. Automat. Control, 57(1):244–249. [doi:10.1109/TAC.2011.2167822]CrossRefMathSciNetGoogle Scholar
  7. Jiang, Y., Jiang, Z.P., 2012. Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics. Automatica, 48(10):2699–2704. [doi:10.1016/j.automatica.2012.06.096]CrossRefMATHMathSciNetGoogle Scholar
  8. Jiang, Z.P., Jiang, Y., 2013. Robust adaptive dynamic programming for linear and nonlinear systems: an overview. Eur. J. Control, 19(5):417–425. [doi:10.1016/j.ejcon.2013.05.017]CrossRefGoogle Scholar
  9. Kurzhanski, A.B., Daryin, A.N., 2008. Dynamic programming for impulse controls. Ann. Rev. Control, 32(2):213–227. [doi:10.1016/j.arcontrol.2008.08.001]CrossRefGoogle Scholar
  10. Lakshmikantham, V., Bainov, D.D., Simeonov, P.S., 1989. Theory of Impulsive Differential Equations. World Scientific, Singapore.CrossRefMATHGoogle Scholar
  11. Lewis, F.L., Vrabie, D., 2009. Reinforcement learning and adaptive dynamic programming for feedback control. IEEE Circ. Syst. Mag., 9(3):32–50. [doi:10.1109/MCAS.2009.933854]CrossRefMathSciNetGoogle Scholar
  12. Liu, B., Teo, K.L., Liu, X.Z., 2008. Optimal control and robust stability of uncertain impulsive dynamical systems. Asian J. Control, 10(3):314–326. [doi:10.1002/asjc.37]CrossRefMathSciNetGoogle Scholar
  13. Liu, D.R., Wei, Q.L., 2013. Finite-approximation-errorbased optimal control approach for discrete-time nonlinear systems. IEEE Trans. Cybern., 43(2):779–789. [doi:10.1109/TSMCB.2012.2216523]CrossRefGoogle Scholar
  14. Liu, X., 1995. Impulsive control and optimization. Appl. Math. Comput., 73(1):77–98. [doi:10.1016/0096-3003(94)00204-H]CrossRefMATHMathSciNetGoogle Scholar
  15. Silva, G.N., Vinter, R.B., 1997. Necessary conditions for optimal impulsive control problems. SIAM J. Control Optim., 35(6):1829–1846. [doi:10.1137/S0363012995281857]CrossRefMATHMathSciNetGoogle Scholar
  16. Vamvoudakis, K.G., Lewis, F.L., 2010. Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem. Automatica, 46(5):878–888. [doi:10.1016/j.automatica.2010.02.018]CrossRefMATHMathSciNetGoogle Scholar
  17. Wang, F.Y., Zhang, H.G., Liu, D.R., 2009. Adaptive dynamic programming: an introduction. IEEE Comput. Intell. Mag., 4(2):39–47. [doi:10.1109/MCI.2009.932261]CrossRefGoogle Scholar
  18. Wang, J.R., Yang, Y.L., 2010. Optimal control of linear impulsive antiperiodic boundary value problem on infinite dimensional spaces. Discr. Dynam. Nat. Soc., Article ID 673013. [doi:10.1155/2010/673013]Google Scholar
  19. Wang, X.H., 2008. Optimal Control of Impulsive Systems Using Adaptive Critic Neural Network. PhD Thesis, Missouri University of Science and Technology, Rolla, Missouri, USA.Google Scholar
  20. Wang, X.H., Balakrishnan, S.N., 2010. Optimal neurocontroller synthesis for impulse-driven systems. Neur. Networks, 23(1):125–134. [doi:10.1016/j.neunet.2009.08.009]CrossRefMATHGoogle Scholar
  21. Wang, X.H., Luo, W.Z., Balakrishnan, S.N., 2012. Linear impulsive system optimization using adaptive dynamic programming. 12th Int. Conf. on Control Automation Robotics and Vision, p.725–730. [doi:10.1109/ICARCV.2012.6485247]Google Scholar
  22. Werbos, P.J., 1974. Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. PhD Thesis, Harvard University, USA.Google Scholar
  23. Werbos, P.J., 2008. Foreword-ADP: the key direction for future research in intelligent control and understanding brain intelligence. IEEE Trans. Syst. Man Cybern. B, 38(4):898–900. [doi:10.1109/TSMCB.2008.924139]CrossRefGoogle Scholar
  24. Werbos, P.J., McAvoy, T., Su, T., 1992. Handbook of Intelligent Control. Van Nostrand Reinhold, New York.Google Scholar
  25. Wu, Z., Zhang, F., 2011. Stochastic maximum principle for optimal control problems of forward-backward systems involving impulse controls. IEEE Trans. Automat. Control, 56(6):1401–1406.CrossRefMathSciNetGoogle Scholar
  26. Yang, T., 1999. Impulsive control. IEEE Trans. Automat. Control, 44(5):1081–1083.CrossRefMATHMathSciNetGoogle Scholar

Copyright information

© Journal of Zhejiang University Science Editorial Office and Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Xiao-hua Wang
    • 1
    • 2
  • Juan-juan Yu
    • 1
  • Yao Huang
    • 1
  • Hua Wang
    • 1
    • 2
  • Zhong-hua Miao
    • 1
    • 2
  1. 1.School of Mechatronics Engineering and AutomationShanghai UniversityShanghaiChina
  2. 2.Shanghai Key Laboratory of Power Station Automation TechnologyShanghai UniversityShanghaiChina

Personalised recommendations