Skip to main content

Advertisement

SpringerLink
Log in

Exploration of the applicability of probabilistic inference for learning control in underactuated autonomous underwater vehicles

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

Underwater vehicles are employed in the exploration of dynamic environments where tuning of a specific controller for each task would be time-consuming and unreliable as the controller depends on calculated mathematical coefficients in idealised conditions. For such a case, learning task from experience can be a useful alternative. This paper explores the capability of probabilistic inference learning to control autonomous underwater vehicles that can be used for different tasks without re-programming the controller. Probabilistic inference learning uses a Gaussian process model of the real vehicle to learn the correct policy with a small number of real field experiments. The use of probabilistic reinforcement learning looks for a simple implementation of controllers without the burden of coefficients calculation, controller tuning or system identification. A series of computational simulations were employed to test the applicability of model-based reinforcement learning in underwater vehicles. Three simulation scenarios were evaluated: waypoint tracking, depth control and 3D path tracking control. The 3D path tracking is done by coupling together a line-of-sight law with probabilistic inference for learning control. As a comparison study LOS-PILCO algorithm can perform better than a robust LOS-PID. The results show that probabilistic model-based reinforcement learning can be a deployable solution to motion control of underactuated AUVs as it can generate capable policies with minimum quantity of episodes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23

Similar content being viewed by others

References

  • Alonge, F., D’Ippolito, F., & Raimondi, F. M. (2001). Trajectory tracking of underactuated underwater vehicles. In Proceedings of the 40th IEEE conference on decision and control (Cat. No.01CH37228) (Vol. 5, pp. 4421–4426). https://doi.org/10.1109/CDC.2001.980898.

  • Ataei, M., & Yousefi-Koma, A. (2015). Three-dimensional optimal path planning for waypoint guidance of an autonomous underwater vehicle. Robotics and Autonomous Systems, 67, 23–32.

    Article  Google Scholar 

  • Bi, S., Niu, C., Cai, Y., Zhang, L., & Zhang, H. (2014). A waypoint-tracking controller for a bionic autonomous underwater vehicle with two pectoral fins. Advanced Robotics, 28(10), 673–681. https://doi.org/10.1080/01691864.2014.888373.

    Article  Google Scholar 

  • Bian, X., Zhou, J., Yan, Z., & Jia, H . (2012). Adaptive neural network control system of path following for AUVs. In 2012 Proceedings of IEEE Southeastcon (pp. 1–5). https://doi.org/10.1109/SECon.2012.6196982.

  • Breivik, M., & Fossen, T. I. (2005). Guidance-based path following for autonomous underwater vehicles. In Oceans 2005 (Vol. 1–3, pp. 2807–2814).

  • Breivik, M., & Fossen, T. I. (2009). Guidance laws for autonomous underwater vehicles, chap 4. In A. V. Inzartsev (Ed.), Underwater vehicles. Rijeka: IntechOpen. https://doi.org/10.5772/6696.

    Chapter  Google Scholar 

  • Caharija, W., Pettersen, K. Y., Gravdahl, J. T., & Borhaug, E. (2012). Path following of underactuated autonomous underwater vehicles in the presence of ocean currents. In 2012 IEEE 51st annual conference on decision and control (CDC) (pp. 528–535).

  • Carlucho, I., Paula, M. D., Wang, S., Petillot, Y., & Acosta, G. G. (2018). Adaptive low-level control of autonomous underwater vehicles using deep reinforcement learning. Robotics and Autonomous Systems, 107, 71–86. https://doi.org/10.1016/j.robot.2018.05.016.

    Article  Google Scholar 

  • Chu, Z., & Zhu, D. (2015). 3d path-following control for autonomous underwater vehicle based on adaptive backstepping sliding mode. In 2015 IEEE international conference on information and automation (pp. 1143–1147). https://doi.org/10.1109/ICInfA.2015.7279458

  • De Paula, M., & Acosta, G. G. (2015). Trajectory tracking algorithm for autonomous vehicles using adaptive reinforcement learning. In OCEANS 2015 - MTS/IEEE Washington (pp. 1–8). https://doi.org/10.23919/OCEANS.2015.7401861.

  • Deisenroth, M., & Rasmussen, C. E. (2011). Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th international conference on machine learning (ICML-11) (pp. 465–472).

  • Deisenroth, M. P., Fox, D., & Rasmussen, C. E. (2015). Gaussian processes for data-efficient learning in robotics and control. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(2), 408–423. https://doi.org/10.1109/TPAMI.2013.218.

    Article  Google Scholar 

  • Do, K. D., Pan, J., & Jiang, Z. P. (2004). Robust and adaptive path following for underactuated autonomous underwater vehicles. Ocean Engineering, 31(16), 1967–1997. https://doi.org/10.1016/j.oceaneng.2004.04.006.

    Article  Google Scholar 

  • Dong, Z., Wan, L., Li, Y., Liu, T., Zhuang, J., & Zhang, G. (2015). Point stabilization for an underactuated auv in the presence of ocean currents. International Journal of Advanced Robotic Systems, 12(7), 100. https://doi.org/10.5772/61037.

    Article  Google Scholar 

  • Duvenaud, D. (2014). Automatic model construction with gaussian processes. Ph.D. thesis., University of Cambridge.

  • Fjerdingen, S. A., Kyrkjebø, E., & Transeth, A. A. (2010). Auv pipeline following using reinforcement learning. In Robotics (ISR), 2010 41st international symposium on and 2010 6th German conference on robotics (ROBOTIK), VDE (pp. 1–8).

  • Forouzantabar, A., Gholami, B., & Azadi, M. (2012). Adaptive neural network control of autonomous underwater vehicles. World Academy of Science, Engineering and Technology, 6(7), 304–309.

    Google Scholar 

  • Fossen, T. I. (1994). Guidance and control of ocean vehicles (Vol. 199). New York: Wiley.

    Google Scholar 

  • Gao, J., Yan, W., Zhao, N., & Xu, D. (2010). Global path following control for unmanned underwater vehicles. In 2010 29th Chinese Control Conference (CCC) (pp. 3188–3192). IEEE.

  • Gaskett, C., Wettergreen, D., & Zelinsky, A., et al. (1999). Reinforcement learning applied to the control of an autonomous underwater vehicle. In Proceedings of the Australian conference on robotics and automation (AuCRA99).

  • Gertler, M., & Hagen, G. R. (1967). Standard equations of motion for submarine simulation. David W Taylor Naval Ship Research and Development Center Bethesda MD: Tech. rep.

  • Holsen, S. A. (2015). Dune: Unified navigation environment for the remus 100 auv-implementation, simulator development, and field experiments. Master’s thesis, NTNU.

  • Kawano, H. (2005). Method for applying reinforcement learning to motion planning and control of under-actuated underwater vehicle in unknown non-uniform sea flow. In 2005 IEEE/RSJ international conference on intelligent robots and systems (pp. 996–1002). https://doi.org/10.1109/IROS.2005.1544973.

  • Kim, J., & Chung, W. K. (2006). Accurate and practical thruster modeling for underwater vehicles. Ocean Engineering, 33(5–6), 566–586.

    Article  Google Scholar 

  • Knudsen, K. B., Nielsen, M. C., & Schjólberg, I. (2019). Deep learning for station keeping of AUVs. In Oceans 2019 MTS/IEEE Seattle (pp. 1–6). https://doi.org/10.23919/OCEANS40490.2019.8962598.

  • Liang, X., You, Y., Su, L., Li, W., & Zhang, J. (2015). Path following control for underactuated auv based on feedback gain backstepping. Technical Gazette, 22(4), 829–835.

    Google Scholar 

  • Liang, X., Wan, L., Blake, J. I., Shenoi, R. A., & Townsend, N. (2016). Path following of an underactuated auv based on fuzzy backstepping sliding mode control. International Journal of Advanced Robotic Systems, 13(3), 122. https://doi.org/10.5772/64065.

    Article  Google Scholar 

  • Prestero, T. (2001). Verification of a six-degree of freedom simulation model for the remus autonomous underwater vehicle. Thesis.

  • Rasmussen, C. E. (2004). Gaussian processes in machine learning. In O. Bousquet, U. von Luxburg, & G. Rätsch (Eds.), Advanced lectures on machine learning. ML 2003. Lecture notes in computer science (Vol. 3176). Berlin, Heidelberg: Springer.

  • Repoulias, F., & Papadopoulos, E. (2007). Planar trajectory planning and tracking control design for underactuated AUVs. Ocean Engineering, 34(11), 1650–1667. https://doi.org/10.1016/j.oceaneng.2006.11.007.

    Article  Google Scholar 

  • Roper, C. (2007). Using the Gavia Auv system to locate and document munitions dumped at sea. Online.

  • Rout, R., & Subudhi, B. (2017). Narmax self-tuning controller for line-of-sight-based waypoint tracking for an autonomous underwater vehicle. IEEE Transactions on Control Systems Technology, 25(4), 1529–1536. https://doi.org/10.1109/TCST.2016.2613969.

    Article  Google Scholar 

  • Saravanakumar, S., & Asokan, T. (2011). Waypoint guidance based planar path following and obstacle avoidance of autonomous underwater vehicle. In ICINCO (2) (pp. 191–198).

  • Shen, H., & Guo, C. (2016). Path-following control of underactuated ships using actor-critic reinforcement learning with MLP neural networks. In 2016 6th international conference on information science and technology (ICIST) (pp. 317–321). https://doi.org/10.1109/ICIST.2016.7483431

  • Sutton, R. S., & Barto, A. G. (2011). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.

    MATH  Google Scholar 

  • Thompson, F., Galeazzi, R., & Guihen, D. (2020). Field trials of an energy-aware mission planner implemented on an autonomous surface vehicle. Journal of Field Robotics,. https://doi.org/10.1002/rob.21942.

    Article  Google Scholar 

  • Wang, J., Wang, C., Wei, Y., & Zhang, C. (2018). Three-dimensional path following of an underactuated auv based on neuro-adaptive command filtered backstepping control. IEEE Access,. https://doi.org/10.1109/ACCESS.2018.2883081.

    Article  Google Scholar 

  • Wang, S., Shen, Y., Sha, Q., Li, G., Jiang, J., Wan, J., Yan, T., & He, B. (2017). Nonlinear path following of autonomous underwater vehicle considering uncertainty. In 2017 IEEE Underwater Technology (UT) (pp. 1–4). https://doi.org/10.1109/UT.2017.7890302.

  • Watkins, C. J. C. H. (1989). Learning from delayed rewards. Ph.D thesis, King’s College, Cambridge.

  • Xiang, X., Lapierre, L., Liu, C., & Jouvencel, B. (2011). Path tracking: Combined path following and trajectory tracking for autonomous underwater vehicles. In 2011 IEEE/RSJ international conference on intelligent robots and systems (pp. 3558–3563). https://doi.org/10.1109/IROS.2011.6094949.

  • Xiang, X., Lapierre, L., & Jouvencel, B. (2015). Smooth transition of auv motion control: From fully-actuated to under-actuated configuration. Robotics and Autonomous Systems, 67, 14–22. https://doi.org/10.1016/j.robot.2014.09.024. (advances in Autonomous Underwater Robotics).

    Article  Google Scholar 

  • Xiang, X., Yu, C., & Zhang, Q. (2017). Robust fuzzy 3d path following for autonomous underwater vehicle subject to uncertainties. Computers and Operations Research, 84, 165–177. https://doi.org/10.1016/j.cor.2016.09.017.

    Article  MathSciNet  MATH  Google Scholar 

  • Yang, X., Shen, Y., Wang, K., Sha, Q., He, B., & Yan, T. (2016). Path following for an autonomous underwater vehicle using gp-los. In OCEANS 2016-Shanghai (pp. 1–5). IEEE

  • Yu, C., Xiang, X., & Dai, J. (2016). 3d path following for under-actuated auv via nonlinear fuzzy controller. In OCEANS 2016 - Shanghai (pp 1–7). https://doi.org/10.1109/OCEANSAP.2016.7485727.

  • Yu, R., Shi, Z., Huang, C., Li, T., & Ma, Q. (2017). Deep reinforcement learning based optimal trajectory tracking control of autonomous underwater vehicle. In 2017 36th Chinese Control Conference (CCC) (pp. 4958–4965). IEEE.

Download references

Acknowledgements

The authors thank Defence Science and Technology Group for the loan of the vehicle MULLAYA to the Australian Maritime College, and constant support on the platform development.

Author information

Authors and Affiliations

  1. Australian Maritime College, University of Tasmania, Newnham, Australia

    Wilmer Ariza Ramirez, Zhi Quan Leong, Hung Duc Nguyen & Shantha Gamini Jayasinghe

Authors
  1. Wilmer Ariza Ramirez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhi Quan Leong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hung Duc Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shantha Gamini Jayasinghe
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wilmer Ariza Ramirez.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ariza Ramirez, W., Leong, Z.Q., Nguyen, H.D. et al. Exploration of the applicability of probabilistic inference for learning control in underactuated autonomous underwater vehicles. Auton Robot 44, 1121–1134 (2020). https://doi.org/10.1007/s10514-020-09922-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-020-09922-z

Keywords

Search

Navigation