Skip to main content

Advertisement

SpringerLink
Log in

Design of Deep Reinforcement Learning Controller Through Data-assisted Model for Robotic Fish Speed Tracking

  • Research Article
  • Published:
Journal of Bionic Engineering Aims and scope Submit manuscript

Abstract

It is common for robotic fish to generate thrust using reactive force generated by the tail’s physical motion, which interacts with the surrounding fluid. The coupling effect of the body strongly correlates with this thrust. However, hydrodynamics cannot be wholly modeled in analytical form. Therefore, data-assisted modeling is necessary for robotic fish. This work presents the first method of its kind using Genetic Algorithm (GA)-based optimization methods for data-assistive modeling for robotic fish applications. To begin, experimental data are collected in real time with the robotic fish that has been designed and fabricated using 3D printing. Then, the model’s influential parameters are estimated using an optimization problem. Further, a model-based deep reinforcement learning (DRL) controller is proposed to track the desired speed through extensive simulation work. In addition to a deep deterministic policy gradient (DDPG), a twin delayed DDPG (TD3) is employed in the training of the RL agent. Unfortunately, due to its local optimization problem, the RL-DDPG controller failed to perform well during training. In contrast, the RL-TD3 controller effectively learns the control policies and overcomes the local optima problem. As a final step, controller performance is evaluated under different disturbance conditions. In contrast to DDPG and GA-tuned proportional-integral controllers, the proposed model with RL-TD3 controller significantly improves the performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Data Availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author upon reasonable request.

References

  1. Verma, S., & Xu, J. X. (2017). Data-assisted modeling and speed control of a robotic fish. IEEE Transactions on Industrial Electronics, 64, 4150–4157.

    Article  Google Scholar 

  2. Li, X., Ren, Q., & Xu, J. X. (2016). Precise speed tracking control of a robotic fish via iterative learning control. IEEE Transactions on Industrial Electronics, 63, 2221–2228.

    Google Scholar 

  3. Yu, J. Z., Yuan, J., Wu, Z. X., & Tan, M. (2016). Data-driven dynamic modeling for a swimming robotic fish. IEEE Transactions on Industrial Electronics, 63, 5632–5640.

    Article  Google Scholar 

  4. Zuo, W., Dhal, K., Keow, A., Chakravarthy, A., & Chen, Z. (2020). Model-based control of a robotic fish to enable 3d maneuvering through a moving orifice. IEEE Robotics and Automation Letters, 5, 4719–4726.

    Article  Google Scholar 

  5. Wang, J., & Tan, X. (2015). Averaging tail-actuated robotic fish dynamics through force and moment scaling. IEEE Transactions on Robotics, 31, 906–917.

    Article  MathSciNet  Google Scholar 

  6. Wang, J., McKinley, P. K., & Tan, X. (2015). Dynamic modeling of robotic fish with a base-actuated flexible tail. Journal of Dynamic Systems, Measurement and Control, Transactions of the ASME, 137, 011004.

    Article  Google Scholar 

  7. Koca, G. O., Bal, C., Korkmaz, D., Bingol, M. C., Ay, M., Akpolat, Z. H., & Yetkin, S. (2018). Three-dimensional modeling of a robotic fish based on real carp locomotion. Applied Sciences (Switzerland), 8, 180.

    Article  Google Scholar 

  8. Suebsaiprom, P., & Lin, C. L. (2015). Maneuverability modeling and trajectory tracking for fish robot. Control Engineering Practice, 45, 22–36.

    Article  Google Scholar 

  9. Hamamci, S., Korkmaz, D., Akpolat, Z. H., Soygüder, S., & Alli, H. (2015). Dynamic simulation model of a biomimetic robotic fish with multi-joint propulsion mechanism. Transactions of the Institute of Measurement and Control, 37, 684–695.

    Article  Google Scholar 

  10. Zhang, S. W., Qian, Y., Liao, P., Qin, F., & Yang, J. M. (2016). Design and control of an agile robotic fish with integrative biomimetic mechanisms. IEEE/ASME Transactions on Mechatronics, 21, 1846–1857.

    Article  Google Scholar 

  11. Yan, S., Wu, Z., Wang, J., Tan, M., & Yu, J. (2020). Efficient cooperative structured control for a multi-joint biomimetic robotic fish. IEEE/ASME Transactions on Mechatronics, 26, 2506–2516.

    Article  Google Scholar 

  12. Korkmaz, D., OzmenKoca, G., Li, G., Bal, C., Ay, M., & Akpolat, Z. H. (2021). Locomotion control of a biomimetic robotic fish based on closed loop sensory feedback CPG model. Journal of Marine Engineering and Technology, 20, 125–137.

    Article  Google Scholar 

  13. Chen, J. Y., Yin, B., Wang, C. C., Xie, F. R., Du, R. X., & Zhong, Y. (2021). Bioinspired closed-loop CPG-based control of a robot fish for obstacle avoidance and direction tracking. Journal of Bionic Engineering, 18, 171–183.

    Article  Google Scholar 

  14. Su, Z., Yu, J. Z., Tan, M., & Zhang, J. (2014). Implementing flexible and fast turning maneuvers of a multijoint robotic fish. IEEE/ASME Transactions on Mechatronics, 19, 329–338.

    Article  Google Scholar 

  15. Yu, J. Z., Tan, M., Wang, S., & Chen, E. K. (2004). Development of a biomimetic robotic fish and its control algorithm. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 34, 1798–1810.

    Article  Google Scholar 

  16. Sun, W. G., Liu, Z. M., Ren, Z. Y., Wang, G., Yuan, T., & Wen, L. (2020). Linear acceleration of an undulatory robotic fish with dynamic morphing median fin under the instantaneous self-propelled condition. Journal of Bionic Engineering, 17, 241–253.

    Article  Google Scholar 

  17. Verma, S., & Xu, J. X. (2018). Analytic modeling for precise speed tracking of multilink robotic fish. IEEE Transactions on Industrial Electronics, 65, 5665–5672.

    Article  Google Scholar 

  18. Suebsaiprom, P., Lin, C. L., & Engkaninan, A. (2017). Undulatory locomotion and effective propulsion for fish-inspired robot. Control Engineering Practice, 58, 66–77.

    Article  Google Scholar 

  19. Zhang, F., Ennasr, O., Litchman, E., & Tan, X. (2015). Autonomous sampling of water columns using gliding robotic fish: Algorithms and harmful-algae-sampling experiments. IEEE Systems Journal, 10, 1271–1281.

    Article  Google Scholar 

  20. Xu, J. X., Niu, X. L., & Guo, Z. Q. Sliding mode control design for a carangiform robotic fish. Proceedings of IEEE International Workshop on Variable Structure Systems, Mumbai, India, 2012, 308–313.

  21. Zhang, P., Wu, Z., Meng, Y., Tan, M., & Yu, J. (2020). Nonlinear model predictive position control for a tail-actuated robotic fish. Nonlinear Dynamics, 101, 2235–2247.

    Article  Google Scholar 

  22. Wen, L., Wang, T., Wu, G., Liang, J., & Wang, C. (2012). Novel method for the modeling and control investigation of efficient swimming for robotic fish. IEEE Transactions on Industrial Electronics, 59, 3176–3188.

    Article  Google Scholar 

  23. Yu, J. Z., Sun, F. H., Xu, D., & Tan, M. H. (2016). Embedded vision-guided 3-d tracking control for robotic fish. IEEE Transactions on Industrial Electronics, 63, 355–363.

    Article  Google Scholar 

  24. Hu, T., Low, K. H., Shen, L., & Xu, X. (2014). Effective phase tracking for bioinspired undulations of robotic fish models: A learning control approach. IEEE/ASME Transactions on Mechatronics, 19, 191–200.

    Article  Google Scholar 

  25. Stearns, H., Fine, B., & Tomizuka, M. Iterative identification of feedforward controllers for iterative learning control. IFAC Proceedings Volumes (IFAC-PapersOnline), Gifu, Japan, 2009, 203–208.

  26. Sedighizadeh, M., & Rezazadeh, A. (2010). A modified adaptive wavelet PID control based on reinforcement learning for wind energy conversion system control. Advances in Electrical and Computer Engineering, 10, 153–159.

    Article  Google Scholar 

  27. Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. Continuous control with deep reinforcement learning. 4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings, San Juan, Puerto Rico, 2016.

  28. Liu, J. C., Liu, Z. N., Wu, Z. X., & Yu, J. Z. (2020) Three-dimensional path following control of an underactuated robotic dolphin using deep reinforcement learning. IEEE International Conference on Real-Time Computing and Robotics, RCAR 2020, Asahikawa, Japan, 315–320.

  29. Fujimoto, S., Van Hoof, H., & Meger, D. Addressing function approximation error in actor-critic methods. 35th International Conference on Machine Learning, ICML 2018, 4. Stockholm, Sweden, 2018, 1587–1596.

  30. Dankwa, S., & Zheng, W. Twin-delayed DDPG: a deep reinforcement learning technique to model a continuous movement of an intelligent robot agent. ACM International Conference Proceeding Series, New York, United States, 2019, 1–5.

  31. Duraisamy, P., Kumar Sidharthan, R., & Nagarajan Santhanakrishnan, M. (2019). Design, modeling, and control of biomimetic fish robot: A review. Journal of Bionic Engineering, 16, 967–993.

    Article  Google Scholar 

  32. Lighthill, M. J. (1960). Note on the swimming of slender fish. Journal of Fluid Mechanics, 9, 305–317.

    Article  MathSciNet  Google Scholar 

  33. Duraisamy, P., & Santhanakrishnan, M. N. (2021). Hydrodynamic modeling and design of robotic fish using slender body theory. IOP Conference Series: Materials Science and Engineering, 1012, 012007.

    Article  Google Scholar 

  34. Szymak, P. (2016). Using neuro-evolutionary-fuzzy method to control a swarm of unmanned underwater vehicles. Control Engineering and Applied Informatics, 18, 82–92.

    Google Scholar 

  35. Oh, B., Na, Y., Yang, J., Park, S., Nang, J., & Kim, J. (2010). Genetic algorithm-based dynamic vehicle route search using car-to-car communication. Advances in Electrical and Computer Engineering, 10, 81–86.

    Article  Google Scholar 

  36. Mendes, W. R., Pereira, F. G., & Cavalieri, D. C. (2018). A hybrid model based on genetic algorithm and space-filling curve applied to optimization of vehicle routes. Advances in Electrical and Computer Engineering, 18, 45–52.

    Article  Google Scholar 

  37. Zuo, X., Xue, H. F., Wang, X. Y., Du, W. R., Tian, T., Gao, S., & Zhang, P. (2021). A deep reinforcement learning method based on deterministic policy gradient for multi-agent cooperative competition. Control Engineering and Applied Informatics, 23, 88–98.

    Google Scholar 

  38. Tiong, T., Saad, I., Teo, K. T. K., & bin Lago, H. Deep reinforcement learning with robust deep deterministic policy gradient. 2nd International Conference on Electrical, Control and Instrumentation Engineering (ICECIE), IEEE, Kuala Lumpur, Malaysia, 2020.

  39. Joohyun, W., Chanwoo, Y., & Nakwan, K. (2019). Deep reinforcement learning-based controller for path following of an unmanned surface vehicle. Ocean Engineering, 183, 155–166.

    Article  Google Scholar 

  40. Stephen D., & Wenfeng Z. Twin-delayed DDPG: a deep reinforcement learning technique to model a continuous movement of an intelligent robot agent. 3rd International Conference on Vision, Image and Signal Processing ICVISP, Vancouver, BC, Canada, 2019.

Download references

Acknowledgements

We would like to thank Rakesh Kumar S for his useful feedback that improved this paper.

Author information

Authors and Affiliations

  1. Robotics Lab, School of EEE, SASTRA Deemed University, Thanjavur, Tamil Nadu, 613401, India

    Palmani Duraisamy & Manigandan Nagarajan Santhanakrishnan

  2. Department of ECE, School of EEE, SASTRA Deemed University, Thanjavur, Tamil Nadu, 613401, India

    Amirtharajan Rengarajan

Authors
  1. Palmani Duraisamy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Manigandan Nagarajan Santhanakrishnan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Amirtharajan Rengarajan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Manigandan Nagarajan Santhanakrishnan.

Ethics declarations

Conflict of Interest

No funding was received to assist with the preparation of this manuscript. Therefore, the authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Duraisamy, P., Nagarajan Santhanakrishnan, M. & Rengarajan, A. Design of Deep Reinforcement Learning Controller Through Data-assisted Model for Robotic Fish Speed Tracking. J Bionic Eng 20, 953–966 (2023). https://doi.org/10.1007/s42235-022-00309-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42235-022-00309-7

Keywords

Search

Navigation