Skip to main content
SpringerLink
Log in

Neural Network-Based Optimal Tracking Control of Continuous-Time Uncertain Nonlinear System via Reinforcement Learning

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

In this note, optimal tracking control for uncertain continuous-time nonlinear system is investigated by using a novel reinforcement learning (RL) scheme. The uncertainty here refers to unknown system drift dynamics. Based on the nonlinear system and reference signal, we firstly formulate the tracking problem by constructing an augmented system. The optimal tracking control problem for original nonlinear system is thus transformed into solving the Hamilton–Jacobi–Bellman (HJB) equation of the augmented system. A new single neural network (NN)-based online RL method is proposed to learn the solution of tracking HJB equation while the corresponding optimal control input that minimizes the tracking HJB equation is calculated in a forward-in-time manner without requiring any value, policy iterations and the system drift dynamics. In order to relax the dependence of the RL method on traditional Persistence of Excitation (PE) conditions, a concurrent learning technique is adopted to design the NN tuning laws. The Uniformly Ultimately Boundedness of NN weight errors and closed-loop augmented system states are rigorous proved. Three numerical simulation examples are given to demonstrate the effectiveness of the proposed scheme.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

References

  1. Lewis FL, Jagannathan S, Yesildirek A (1998) Neural network control of robot manipulators and nonlinear systems. Taylor & Francis, Philadelphia, PA

    Google Scholar 

  2. Mahony R, Hamel T (2004) Robust trajectory tracking for a scale model autonomous helicopter. Int J Robust Nonlinear Control 14(12):1035

    Article  MathSciNet  Google Scholar 

  3. Huang J, Wen C, Wang W, Jiang ZP (2014) Adaptive output feedback tracking control of a nonholonomic mobile robot. Automatica 50(3):821

    Article  MathSciNet  Google Scholar 

  4. Tang X, Tao G, Joshi SM (2003) Adaptive actuator failure compensation for parametric strict feedback systems and an aircraft application. Automatica 39(11):1975

    Article  MathSciNet  Google Scholar 

  5. Lewis FL, Vrabie DL, Syrmos VL (2015) Optimal control, 3rd edn. Wiley, New York

    MATH  Google Scholar 

  6. Mannava A, Balakrishnan SN, Tang L, Landers RG (2012) Optimal tracking control of motion systems. IEEE Trans Control Syst Technol 20(6):1548

    Article  Google Scholar 

  7. Sharma R, Tewari A (2013) Optimal nonlinear tracking of spacecraft attitude maneuvers. IEEE Trans Control Syst Technol 12(5):677

    Article  Google Scholar 

  8. Liu T, Liang S, Xiong Q, Wang K (2018) Adaptive critic based optimal neurocontrol of a distributed microwave heating system using diagonal recurrent network. IEEE Access 6:68839

    Article  Google Scholar 

  9. Liu T, Liang S, Xiong Q, Wang K (2019) Data-based online optimal temperature tracking control in continuous microwave heating system by adaptive dynamic programming. Neural Process Lett. https://doi.org/10.1007/s11063-019-10081-1

    Article  Google Scholar 

  10. Sutton R, Barto A (2018) Reinforcement learning: an introduction. The MIT Press, Cambridge

    MATH  Google Scholar 

  11. Lewis FL, Liu D (2015) Reinforcement learning and approximate dynamic programming for feedback control. IEEE Circuits Syst Mag 9(3):32

    Article  Google Scholar 

  12. Ren L, Zhang G, Mu C (2019) Optimal output feedback control of nonlinear partially-unknown constrained-input systems using integral reinforcement learning. Neural Process Lett. https://doi.org/10.1007/s11063-019-10072-2

    Article  Google Scholar 

  13. Qiao L, Wei Q, Liu D (2017) A novel optimal tracking control scheme for a class of discrete-time nonlinear systems using generalised policy iteration adaptive dynamic programming algorithm. Int J Syst Sci 48(3):525

    Article  MathSciNet  Google Scholar 

  14. Zhang H, Wei Q, Luo Y (2008) A novel infinite-time optimal tracking control scheme for a class of discrete-time nonlinear systems via the greedy HDP iteration algorithm. IEEE Trans Syst Man Cybern Part B 38(4):937

    Article  Google Scholar 

  15. Zhang H, Cui L, Zhang X, Luo Y (2011) Data-driven robust approximate optimal tracking control for unknown general nonlinear systems using adaptive dynamic programming method. IEEE Trans Neural Netw 22(12):2226

    Article  Google Scholar 

  16. Xiong Y, Liu D, Ding W (2014) Reinforcement learning for adaptive optimal control of unknown continuous-time nonlinear systems with input constraints. Int J Control 87(3):553

    Article  MathSciNet  Google Scholar 

  17. Kamalapurkar R, Andrews L, Walters P, Dixon WE (2017) Model-based reinforcement learning for infinite-horizon approximate optimal tracking. IEEE Trans Neural Netw Learn Syst 28(3):753

    Article  Google Scholar 

  18. Kiumarsi B, Lewis FL (2017) Actor-critic-based optimal tracking for partially unknown nonlinear discrete-time systems. IEEE Trans Neural Netw Learn Syst 26(1):140

    Article  MathSciNet  Google Scholar 

  19. Modares H, Lewis FL (2014) Optimal tracking control of nonlinear partially-unknown constrained-input systems using integral reinforcement learning. Automatica 50(7):1780

    Article  MathSciNet  Google Scholar 

  20. Modares H, Lewis FL (2014) Linear quadratic tracking control of partially-unknown continuous-time systems using reinforcement learning. IEEE Trans Autom Control 59(11):3051

    Article  MathSciNet  Google Scholar 

  21. Kiumarsi-Khomartash B, Lewis FL, Naghibi-Sistani M, Karimpour A (2013) Optimal tracking control for linear discrete-time systems using reinforcement learning. In: 52nd IEEE Conference on Decision and Control, Florence, pp 3845–3850

  22. Kiumarsi B, Lewis FL, Naghibi-Sistani MB, Karimpour A (2015) Optimal tracking control of unknown discrete-time linear systems using input–output measured data. IEEE Trans Cybern 45(12):2770

    Article  Google Scholar 

  23. Wei Q, Liu D (2014) Adaptive dynamic programming for optimal tracking control of unknown nonlinear systems with application to coal gasification. IEEE Trans Autom Sci Eng 11(4):1020

    Article  MathSciNet  Google Scholar 

  24. Lin X, Qiang D, Kong W, Song C, Huang Q (2015) Adaptive dynamic programming-based optimal tracking control for nonlinear systems using general value iteration

  25. Zhang H, Song R, Wei Q, Zhang T (2011) Optimal tracking control for a class of nonlinear discrete-time systems with time delays based on heuristic dynamic programming. IEEE Trans Neural Netw 22(12):1851

    Article  Google Scholar 

  26. Gao W, Jiang Z (2016) Adaptive dynamic programming and adaptive optimal output regulation of linear systems. IEEE Trans Autom Control 61(12):4164

    Article  MathSciNet  Google Scholar 

  27. Han KZ, Jian F, Cui X (2017) Fault-tolerant optimised tracking control for unknown discrete-time linear systems using a combined reinforcement learning and residual compensation methodology. Int J Syst Sci 48(13):2811

    Article  MathSciNet  Google Scholar 

  28. Zhang H, Wei Q, Luo Y (2008) A novel infinite-time optimal tracking control scheme for a class of discrete-time nonlinear systems via the greedy HDP iteration algorithm. IEEE Trans Syst Man Cybern Part B (Cybern) 38(4):937

    Article  Google Scholar 

  29. Liu C, Zhang H, Ren H, Liang Y (2019) An analysis of IRL-based optimal tracking control of unknown nonlinear systems with constrained input. Neural Process Lett. https://doi.org/10.1007/s11063-019-10029-5

    Article  Google Scholar 

  30. Bertsekas DP (2005) Dynamic programming and optimal control, 3rd edn. Athena Scientific, Belmont, MA

    MATH  Google Scholar 

  31. Bruce FA (1990) The method of weighted residuals and variational principles. Academic Press, New York

    Google Scholar 

  32. Abu-Khalaf M, Lewis FL (2005) Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach. Automatica 41(5):779

    Article  MathSciNet  Google Scholar 

  33. Vamvoudakis KG, Lewis FL (2010) Online actorcritic algorithm to solve the continuous-time infinite horizon optimal control problem. Automatica 46(5):878

    Article  MathSciNet  Google Scholar 

  34. Luy NT (2014) Reinforecement learning-based optimal tracking control for wheeled mobile robot. Trans Inst Meas Control 36(7):171

    Article  Google Scholar 

  35. Zargarzadeh H, Dierks T, Jagannathan S (2015) Optimal control of nonlinear continuous-time systems in strict-feedback form. IEEE Trans Neural Netw Learn Syst 26(10):2535

    Article  MathSciNet  Google Scholar 

  36. Vamvoudakis KG (2017) Q-learning for continuous-time linear systems: a model-free infinite horizon optimal control approach. Syst Control Lett 100(Complete):14

    Article  MathSciNet  Google Scholar 

  37. Modares H, Lewis FL, Naghibi-Sistani MB (2014) Integral reinforcement learning and experience replay for adaptive optimal control of partially-unknown constrained-input continuous-time systems. Automatica 50(1):193

    Article  MathSciNet  Google Scholar 

  38. Chowdhary G, Johnson E (2010) Concurrent learning for convergence in adaptive control without persistency of excitation. In: 49th IEEE Conference on Decision and Control (CDC), Atlanta, GA, pp 3674–3679

  39. Vamvoudakis KG, Mojoodi A, Ferraz H (2017) Eventtriggered optimal tracking control of nonlinear systems. Int J Robust Nonlinear Control 27(4):598–619

    Article  Google Scholar 

  40. Yang X, Liu D, Wei Q, Wang D (2016) Guaranteed cost neural tracking control for a class of uncertain nonlinear systems using adaptive dynamic programming. Neurocomputing 198:80

    Article  Google Scholar 

Download references

Acknowledgements

This paper is funded by International Graduate Exchange Program of Beijing Institute of Technology.

Author information

Authors and Affiliations

  1. School of Automation, Beijing Institute of Technology, No. 5 Zhongguancun South Street, Haidian District, Beijing, 100081, People’s Republic of China

    Jingang Zhao

Authors
  1. Jingang Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jingang Zhao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, J. Neural Network-Based Optimal Tracking Control of Continuous-Time Uncertain Nonlinear System via Reinforcement Learning. Neural Process Lett 51, 2513–2530 (2020). https://doi.org/10.1007/s11063-020-10220-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-020-10220-z

Keywords

Search

Navigation