Skip to main content
SpringerLink
Log in

Adaptive Dynamic Programming in the Hamiltonian-Driven Framework

  • Chapter
  • First Online:
Handbook of Reinforcement Learning and Control

Part of the book series: Studies in Systems, Decision and Control ((SSDC,volume 325))

  • 7388 Accesses

Abstract

This chapter presents a Hamiltonian-driven framework of adaptive dynamic programming (ADP) for continuous-time nonlinear systems. Three fundamental problems for solving the optimal control problem are presented, i.e., the evaluation of given admissible policy, the comparison between two different admissible policies with respect to the performance, and the performance improvement of given admissible control. It is shown that the Hamiltonian functional can be viewed as the temporal difference for dynamical systems in continuous time. Therefore, the minimization of the Hamiltonian functional is equivalent to the value function approximation. An iterative algorithm starting from an arbitrary admissible control is presented for the optimal control approximation with its convergence proof. The Hamiltonian-driven ADP algorithm can be implemented using a critic only structure, which is trained to approximate the optimal value gradient. Simulation example is conducted to verify the effectiveness of Hamiltonian-driven ADP.

This work was supported in part by the National Natural Science Foundation of China under Grant No. 61903028, in part by the China Post-Doctoral Science Foundation under Grant 2018M641197, in part by the Fundamental Research Funds for the Central Universities under grant No. FRF-TP-18-031A1 and No. FRF-BD-17-002A, in part by the DARPA/Microsystems Technology Office and in part by the Army Research Laboratory under grant No. W911NF-18-2-0260. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 249.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 249.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Abu-Khalaf, M., Lewis, F.L.: Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network hjb approach. Automatica 41(5), 779–791 (2005)

    Google Scholar 

  2. Bellman, R.: Dynamic Programming. Princeton University Press, Princeton, NJ, USA (1957)

    MATH  Google Scholar 

  3. Dimitri, P.: Bertsekas. Nonlinear programming, Athena Scientific, Belmont, MA (1995)

    Google Scholar 

  4. Bertsekas, D.P.: Dynamic programming and optimal control, 3rd edn., vol. 1, Athena Scientific, Belmont, MA (2007)

    Google Scholar 

  5. Fairbank, M., Alonso, E.: Value-gradient learning. In: The 2012 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2012)

    Google Scholar 

  6. Hadda, W., Chellaboina, V.: Nonlinear dynamical systems and control (2008)

    Google Scholar 

  7. Jiang, Yu., Jiang, Z.-P.: Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics. Automatica 48(10), 2699–2704 (2012)

    Article  MathSciNet  Google Scholar 

  8. Orrin Frink, J.R.: Differentiation of sequences. Bull. Am. Math. Soc. 41, 553–560 (1935)

    Article  MathSciNet  Google Scholar 

  9. Kiumarsi, B., Vamvoudakis, K.G., Modares, H., Lewis, F.L.: Optimal and autonomous control using reinforcement learning: a survey. IEEE Trans. Neural Netw. Learn. Syst. 29(6), 2042–2062 (2018)

    Article  MathSciNet  Google Scholar 

  10. Kiumarsi, B., Lewis, F.L., Jiang, Z.-P.: H\(_\infty \) control of linear discrete-time systems: Off-policy reinforcement learning. Automatica 78, 144–152 (2017)

    Article  MathSciNet  Google Scholar 

  11. Lewis, F.L., Liu, D.: Reinforcement learning and approximate dynamic programming for feedback control, vol. 17. Wiley, Hoboken (2013)

    Google Scholar 

  12. Lewis, F.L., Vrabie, D.: Reinforcement learning and adaptive dynamic programming for feedback control. Circuits Syst. Mag. IEEE 9(3), 32–50 (2009)

    Google Scholar 

  13. Lewis, F.L., Vrabie, D., Vamvoudakis, K.G.: Reinforcement learning and feedback control: using natural decision methods to design optimal adaptive controllers. IEEE Control Syst. 32(6), 76–105 (2012)

    Google Scholar 

  14. Daniel, L.: Calculus of variations and optimal control theory: a concise introduction. Princeton University Press, Princeton (2012)

    Google Scholar 

  15. Biao, L., Wu, H.-N., Tingwen, H.: Off-policy reinforcement learning for control design. IEEE Trans. Cybern. 45(1), 65–76 (2015)

    Google Scholar 

  16. Miller, W.T., Werbos, P.J., Sutton, R.S.: Neural networks for control. MIT press, Cambridge (1995)

    Google Scholar 

  17. Modares, H., Lewis, F.L.: Optimal tracking control of nonlinear partially-unknown constrained-input systems using integral reinforcement learning. Automatica 50(7), 1780–1792 (2014)

    Google Scholar 

  18. Modares, H., Lewis, F.L., Jiang, Z.-P.: Tracking control of completely unknown continuous-time systems via off-policy reinforcement learning. IEEE Trans. Neural Netw. Learn. Syst. 26(10), 2550–2562 (2015)

    Google Scholar 

  19. Modares, H., Lewis, F.L., Naghibi-Sistani, M.-B.: Integral reinforcement learning and experience replay for adaptive optimal control of partially-unknown constrained-input continuous-time systems. Automatica 50(1), 193–202 (2014)

    Google Scholar 

  20. Murray, J.J., Cox, C.J., Lendaris, G.G., Saeks, R.: Adaptive dynamic programming. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 32(2), 140–153 (2002)

    Google Scholar 

  21. Padhi, R., Unnikrishnan, N., Wang, X., Balakrishnan, S.N.: A single network adaptive critic (snac) architecture for optimal control synthesis for a class of nonlinear systems. Neural Netw. 19(10), 1648–1660 (2006)

    Google Scholar 

  22. Lev Semenovich Pontryagin.: Mathematical theory of optimal processes. CRC Press, Boca Raton (1987)

    Google Scholar 

  23. Prokhorov, D.V., Wunsch, D.C., et al.: Adaptive critic designs. Neural Netw. IEEE Trans. 8(5), 997–1007 (1997)

    Google Scholar 

  24. Saridis, G.N., Lee, C.-S.G.: An approximation theory of optimal control for trainable manipulators. IEEE Trans. Syst. Man Cybern. 9(3), 152–159 (1979)

    Google Scholar 

  25. Speyer, J.L., Jacobson, D.H.: Primer on optimal control theory, vol. 20. SIAM, Philadelphia (2010)

    Google Scholar 

  26. Sutton, R.S.: Learning to predict by the methods of temporal differences. Mach. Learn. 3(1), 9–44 (1988)

    Google Scholar 

  27. Sutton, R.S., Barto, A.G.: Reinforcement learning: an introduction, vol. 1, MIT Press, Cambridge (1998)

    Google Scholar 

  28. Vamvoudakis, K.G., Ferraz, H.: Model-free event-triggered control algorithm for continuous-time linear systems with optimal performance. Automatica 87, 412–420 (2018)

    Google Scholar 

  29. Vamvoudakis, K.G., Lewis, F.L.: Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem. Automatica 46(5), 878–888 (2010)

    Google Scholar 

  30. Van der Schaft, A.J.: L2-gain analysis of nonlinear systems and nonlinear state-feedback h control. IEEE Trans. Autom. Control 37(6), 770–784 (1992)

    Google Scholar 

  31. Vrabie, D., Lewis, F.: Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems. Neural Netw. 22(3), 237–246 (2009)

    Article  Google Scholar 

  32. Wang, F.-Y., Zhang, H., Liu, D.: Adaptive dynamic programming: an introduction. IEEE Comput. Intell. Mag. 4(2), 39–47 (2009)

    Article  Google Scholar 

  33. Werbos, P.J.: Beyond regression: new tools for prediction and analysis in the behavioral sciences. Ph.D. thesis, Harvard (1974)

    Google Scholar 

  34. Paul John Werbos: Consistency of hdp applied to a simple reinforcement learning problem. Neural Netw. 3(2), 179–189 (1990)

    Article  Google Scholar 

  35. Paul John Werbos: Approximate dynamic programming for real-time control and neural modeling. Handb. Intell. Control.: Neural Fuzzy Adapt. Approaches 15, 493–525 (1992)

    Google Scholar 

  36. Werbos, P.J.: The roots of backpropagation: from ordered derivatives to neural networks and political forecasting, vol. 1. Wiley, Hoboken (1994)

    Google Scholar 

  37. White, D.A., Sofge, D.A.: Handbook of intelligent control: neural, fuzzy, and adaptative approaches. Van Nostrand Reinhold Company, New York (1992)

    Google Scholar 

  38. Yang, Y., Guo, Z.-S., Haoyi Xiong, Z.-S., Ding, D.-W., Yin, Y., Wunsch, D.C.: Data-driven robust control of discrete-time uncertain linear systems via off-policy reinforcement learning. IEEE Trans. Neural Netw. Learn. Syst. (2019)

    Google Scholar 

  39. Yang, Y., Modares, H., Wunsch, D.C., Yin, Y.: Leader-follower output synchronization of linear heterogeneous systems with active leader using reinforcement learning. IEEE Trans. Neural Netw. Learn. Syst. 29(6), 2139–2153 (2018)

    Article  MathSciNet  Google Scholar 

  40. Yang, Y.: Optimal containment control of unknown heterogeneous systems with active leaders. IEEE Trans. Control Syst. Technol. 27(3), 1228–1236 (2019)

    Article  Google Scholar 

  41. Yang, Y., Vamvoudakis, K.G., Ferraz, H., Modares, H.: Dynamic intermittent Q-learning-based model-free suboptimal co-design of L\(_2\)-stabilization. Int. J. Robust Nonlinear Control 29(9), 2673–2694 (2019)

    Article  Google Scholar 

  42. Yang, Y., Wunsch, D.C., Yin, Y.: Hamiltonian-driven adaptive dynamic programming for continuous nonlinear dynamical systems. IEEE Trans. Neural Netw. Learn. Syst. 28(8), 1929–1940 (2017)

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Automation and Electrical Engineering, University of Macau, Avenida da Universidade, Taipa, Macau, China

    Yongliang Yang

  2. School of Automation and Electrical Engineering, University of Science and Technology Beijing, No. 30 Xueyuan Road, Beijing, 100083, China

    Yixin Yin

  3. Department of Electrical and Computer Engoneering, Missouri University of Science and Technology, 301 W. 16th St., Rolla, MO, 65409, USA

    Donald C. Wunsch II

Authors
  1. Yongliang Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Donald C. Wunsch II
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yixin Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yongliang Yang .

Editor information

Editors and Affiliations

  1. The Daniel Guggenheim School of Aerospace Engineering, Georgia Institute of Technology, Atlanta, GA, USA

    Kyriakos G. Vamvoudakis

  2. Department of Electrical Engineering, The University of Texas at Arlington, Arlington, TX, USA

    Yan Wan

  3. Department of Electrical Engineering, The University of Texas at Arlington, Arlington, TX, USA

    Frank L. Lewis

  4. Army Research Office, Durham, NC, USA

    Derya Cansever

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Yang, Y., Wunsch II, D.C., Yin, Y. (2021). Adaptive Dynamic Programming in the Hamiltonian-Driven Framework. In: Vamvoudakis, K.G., Wan, Y., Lewis, F.L., Cansever, D. (eds) Handbook of Reinforcement Learning and Control. Studies in Systems, Decision and Control, vol 325. Springer, Cham. https://doi.org/10.1007/978-3-030-60990-0_7

Download citation

Publish with us

Policies and ethics

Search

Navigation