Skip to main content
SpringerLink
Log in

Robust Optimal Control Based on the Off-Policy Integral Reinforcement Learning Algorithm for Surface Vessel Systems with Unknown Dynamics

  • Conference paper
  • First Online:
Proceedings of the International Conference on Advanced Mechanical Engineering, Automation, and Sustainable Development 2021 (AMAS2021) (AMAS 2021)

Part of the book series: Lecture Notes in Mechanical Engineering ((LNME))

Abstract

This paper develops a robust optimal controller that is applied for fully actuated Surface Vessel Systems with completely unknown dynamics and external disturbance. The new cascade structure of the off-policy integral reinforcement learning (Off-policy IRL) algorithm and kinematic controller. The critic-actor-disturbance neural networks (NN) are built to solve Hamilton-Jacobi-Isaac (HJI) equation in robust optimal control terms. Since a surface vessel is decoupled by the kinematic sub-system and dynamic sub-system, the cascade control system is ideal for obtaining the tracking problem. The convergence of the proposed algorithm to the solution to the tracking HJI equation is shown. Finally, the simulation model is built for the presented method to verify the correctness and effectiveness of the proposed scheme. The simulation results show that the introduced controller gives good performances even that the desired trajectory is complicated and the working condition is hard.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Xiao, B., Yang, X., Huo, X.: A novel disturbance estimation scheme for formation control of ocean surface vessels. IEEE Trans. Industr. Electron. 64(6), 4994–5003 (2017)

    Article  Google Scholar 

  2. Dai, S.L., Wang, M., Wang, C.: Neural learning control of marine surface vessels with guaranteed transient tracking performance. IEEE Trans. Industr. Electron. 63(3), 1717–1727 (2016)

    Article  Google Scholar 

  3. Park, B.S., Kwon, J.-W., Kim, H.: Neural network-based output feedback control for reference tracking of underactuated surface vessels. Automatica 77, 353–359 (2017)

    Google Scholar 

  4. Wang, N., Su, S.F., Pan, X., Yu, X., Xie, G.: Yaw-guided trajectory tracking control of an asymmetric underactuated surface vehicle. IEEE Trans. Ind. Inform. 15(6), 3502–3513 (2019)

    Google Scholar 

  5. Xie, W., Ma, B., Huang, W., Zhao, Y.: Global trajectory tracking control of underactuated surface vessels with non-diagonal inertial and damping matrices. Nonlinear Dyn. 92(4), 1481–1492 (2018). https://doi.org/10.1007/s11071-018-4141-1

    Article  MATH  Google Scholar 

  6. Wang, N., Xie, G., Pan, X., Su, S.F.: Full-State regulation control of asymmetric underactuated surface vehicles. IEEE Trans. Ind. Electron. 66(11), 8741–8750 (2019)

    Article  Google Scholar 

  7. Li, J.-W.: Robust adaptive control of underactuated ships with input saturation. Int. J. Control, 1–10 (2019)

    Google Scholar 

  8. Qin, H., Li, C., Sun, Y., Li, X., Du, Y., Deng, Z.: Finite-time trajectory tracking control of unmanned surface vessel with error constraints and input saturations. J. Franklin Inst. 357(16), 11472–11495 (2020)

    Article  MathSciNet  Google Scholar 

  9. Zhang, J., Yu, S., Yan, Y.: Fixedtime output feedback trajectory tracking control of marine surface vessels subject to unknown external disturbances and uncertainties. ISA Trans. 93, 145–155 (2019)

    Google Scholar 

  10. Zhang, J., Yu, S., Yan, Y.: Fixedtime velocity-free sliding mode tracking control for marine surface vessels with uncertainties and unknown actuator faults. Ocean Eng. 201 (2020)

    Google Scholar 

  11. Van, M.: An enhanced tracking control of marine surface vessels based on adaptive integral sliding mode control and disturbance observer. ISA Trans. 90, 30–40 (2019)

    Article  Google Scholar 

  12. Van, M.: Adaptive neural integral sliding-mode control for tracking control of fully actuated uncertain surface vessels. Int. J. Robust Nonlinear Control 29(5), 1537–1557 (2019)

    Article  MathSciNet  Google Scholar 

  13. Wang, N., Karimi, H.R., Li, H., Su, S.F.: Accurate trajectory tracking of disturbed surface vehicles: a finite-time control approach. IEEE/ASME Trans. Mechatron. 24(3), 1064–1074 (2019)

    Article  Google Scholar 

  14. Zhu, Y., Zhao, D., Li, X.: Using reinforcement learning techniques to solve continuous-time non-linear optimal tracking problem without system dynamics. IET Control Theory Appl. 10(12), 1339–1347 (2016)

    Article  MathSciNet  Google Scholar 

  15. Yin, Z., He, W., Sun, C., Li, G., Yang, C.: Adaptive control of a marine vessel based on reinforcement learning. In: Chinese Control Conference, CCC, vol. 2018, pp. 2735–2740 (2018)

    Google Scholar 

  16. Yin, Z., He, W., Yang, C., Sun, C.: Control design of a marine vessel system using reinforcement learning. Neurocomputing 311, 353–362 (2018)

    Article  Google Scholar 

  17. Wen, G., Ge, S.S., Chen, C.L.P., Tu, F., Wang, S.: Adaptive tracking control of surface vessel using optimized backstepping technique. IEEE Trans. Cybern. 49(9), 3420–3431 (2019)

    Article  Google Scholar 

  18. Zheng, Z., Ruan, L., Zhu, M., Guo, X.: Reinforcement learning control for underactuated surface vessel with output error constraints and uncertainties. Neurocomputing 399, 479–490 (2020)

    Article  Google Scholar 

  19. Walters, P., Kamalapurkar, R., Voight, F., Schwartz, E.M., Dixon, W.E.: Online approximate optimal station keeping of a marine craft in the presence of an irrotational current. IEEE Trans. Robot. 34(2), 486–496 (2018)

    Article  Google Scholar 

  20. Guo, X., Yan, W., Cui, R.: integral reinforcement learning-based adaptive NN control for continuous-time nonlinear MIMO systems with unknown control directions. IEEE Trans. Syst. Man Cybern. Syst., 1–10 (2019)

    Google Scholar 

  21. Martinsen, A.B., Lekkas, A.M., Gros, S., Glomsrud, J.A., Pedersen, T.A.: Reinforcement learning-based tracking control of USVs in varying operational conditions. Front. Robot. AI 7 (2020)

    Google Scholar 

  22. Fossen, T.I.: Marine control system-guidance, navigation and control of ships, rigs and underwater vehicles. J. Guid. Control. Dyn. 28(3), 574–575 (2002)

    MathSciNet  Google Scholar 

  23. Kamalapurkar, R., Dinh, H., Bhasin, S., Dixon, W.E.: Approximate optimal trajectory tracking for continuous-time nonlinear systems. Automatica 51, 40–48 (2015)

    Article  MathSciNet  Google Scholar 

  24. Wu, H.N., Luo, B.: Neural network based online simultaneous policy update algorithm for solving the HJI equation in nonlinear H\infty control. IEEE Trans. Neural Netw. Learn. Syst. 23(12), 1884–1895 (2012)

    Article  Google Scholar 

  25. Modares, H., Lewis, F.L., Jiang, Z.P.: H∞ tracking control of completely unknown continuous-time systems via off-policy reinforcement learning. IEEE Trans. Neural Netw. Learn. Syst. 26(10), 2550–2562 (2015)

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Electrical Engineering, Hanoi University of Science and Technology, Hanoi, Vietnam

    Van Tu Vu, Phuong Nam Dao & Xuan Minh Phan

  2. Faculty of Electrical and Mechanical Engineering, Hai Phong University, Haiphong, Vietnam

    Van Tu Vu

Authors
  1. Van Tu Vu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Phuong Nam Dao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xuan Minh Phan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Van Tu Vu .

Editor information

Editors and Affiliations

  1. Vietnam Association for Science Editing, Hanoi University of Science and Technology, Hanoi, Vietnam

    Banh Tien Long

  2. Department of Materials Science and Engineering, Inha University, Nam-gu, Korea (Republic of)

    Hyung Sun Kim

  3. School of Mechanical Engineering, Nagaoka University of Technology, Nagaoka, Japan

    Kozo Ishizaki

  4. Hanoi University of Science and Technology, Hanoi, Vietnam

    Nguyen Duc Toan

  5. Southern Federal University, Rostov-on-Don, Russia

    Ivan A. Parinov

  6. Department of Mechanical Engineering, Korea Maritime and Ocean University, Busan, Korea (Republic of)

    Yun-Hea Kim

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Vu, V.T., Dao, P.N., Phan, X.M. (2022). Robust Optimal Control Based on the Off-Policy Integral Reinforcement Learning Algorithm for Surface Vessel Systems with Unknown Dynamics. In: Long, B.T., Kim, H.S., Ishizaki, K., Toan, N.D., Parinov, I.A., Kim, YH. (eds) Proceedings of the International Conference on Advanced Mechanical Engineering, Automation, and Sustainable Development 2021 (AMAS2021). AMAS 2021. Lecture Notes in Mechanical Engineering. Springer, Cham. https://doi.org/10.1007/978-3-030-99666-6_124

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-99666-6_124

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-99665-9

  • Online ISBN: 978-3-030-99666-6

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation