Abstract
To ensure the steady navigation for robot stable controls are one of the basic requirements. Control values selection is highly environment dependent. To ensure reusability of control parameter, system needs to generalize over the environment. Adding adaptability in robots to perform effectively in the environments with no prior knowledge reinforcement leaning is a promising approach. However, tuning hyper parameters and attaining correlation between state space and reward function to train a stable reinforcement learning agent is a challenge. This paper is focused, to design a continuous reward function to minimize the sparsity and stabilizes the policy convergence, to attain control generalization for differential drive robot. To achieve that, Twin Delayed Deep Deterministic Policy Gradient is implemented on PyBullet Racecar model in Open-AIGym environment. System was trained to achieve smart primitive control policy, moving forward in the direction of goal by maintaining an appropriate distance from walls to avoid collision. Resulting policy was tested on unseen environments including dynamic goal environment, boundary free environment and continuous path environment on which it outperformed Deep Deterministic Policy Gradient.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Cooper S, Di Fava A, Vivas C, Marchionni L, Ferro F (2020) Ari: the social assistive robot and companion. 2020 29th IEEE International conference on robot and human interactive communication (RO-MAN), pp 745–751 https://doi.org/10.1109/RO-MAN47096.2020.9223470
Rubio F, Valero F, Llopis-Albert C (2019) A review of mobile robots: Concepts, methods, theoretical framework, and applications. Int J Adv Rob Syst 16(2):1729881419839596. https://doi.org/10.1177/1729881419839596
Kormushev P, Calinon S, Caldwell DG (2013) Reinforcement learning in robotics: applications and real-world challenges. Robotics 2(3):122–148. https://doi.org/10.3390/robotics2030122
Nguyen-Tuong D, Peters J (2011) Model learning for robot control: a survey. Cognitive Proc 12:319–340. https://doi.org/10.1007/s10339-011-0404-1
Ugurlu HI, Kalkan S, Saranli A (2021) Reinforcement learning versus conventional control for controlling a planar bi-rotor platform with tail appendage. J Intell Robot Syst 102:1–17. https://doi.org/10.1007/s10846-021-01412-3
Bledt G, Powell MJ, Katz B, Di Carlo J, Wensing PM, Kim S (2018) Mit cheetah 3: Design and control of a robust, dynamic quadruped robot. IEEE/RSJ International Conference on Intelligent Robots and Systems IROS, pp 2245–2252. https://doi.org/10.1109/IROS.2018.8593885
Haarnoja T, Ha S, Zhou A, Tan J, Tucker G, Levine S (2019) Learning to walk via deep reinforcement learning. Robotics: Sci Syst https://doi.org/10.15607/RSS.2019.XV.011
Abo Mosali N, Shamsudin SS, Alfandi O, Omar R, Al-Fadhali N (2022) Twin delayed deep deterministic policy gradient-based target tracking for unmanned aerial vehicle with achievement rewarding and multistage training. IEEE Access 10:23545–23559. https://doi.org/10.1109/ACCESS.2022.3154388
Fujimoto S, Hoof H, Meger D (2018) Addressing function approximation error in actor-critic methods. 35th International conference on machine learning 80, pp 1587–1596
Lillicrap TP, Hunt JJ, Pritzel A, Heess N, Erez T, Tassa Y, Silver D, Wierstra D (2016) Continuous control with deep reinforcement learning. 4th International conference on learning representations (ICLR)
Xu X, Chen Y, Bai C (2021) Deep reinforcement learning-based accurate control of planetary soft landing. Sensors 21(23):8161. https://doi.org/10.3390/s21238161
Pérez-Gil Ó, Barea R, López-Guillén E, Bergasa LM, Gomez-Huelamo C, Gutiérrez R, Diaz-Diaz A (2022) Deep reinforcement learning based control for autonomous vehicles in Carla. Multimed Tools Appl 81(3):3553–3576. https://doi.org/10.1007/s11042-021-11437-3
Dai H, Chen P, Yang H (2022) Driving torque distribution strategy of skid-steering vehicles with knowledge-assisted reinforcement learning. Appl Sci 12(10):5171. https://doi.org/10.3390/app12105171
Jin L, Tian D, Zhang Q, Wang J (2020) Optimal torque distribution control of multi-axle electric vehicles with in-wheel motors based on DDPG algorithm. Energies 13(6):1331. https://doi.org/10.3390/en13061331
Chen Y, Han W, Zhu Q, Liu Y, Zhao J (2022) Target-driven obstacle avoidance algorithm based on DDPG for connected autonomous vehicles. EURASIP J Adv Signal Proc 2022(1):1–22. https://doi.org/10.1186/s13634-022-00872-5
Konda V, Tsitsiklis J (1999) Actor-critic algorithms. Advances in Neural Information Processing Systems 12
Zhou W, Li W (2022) Programmatic reward design by example. 36th AAAI Conference on Artificial Intelligence, 36(8), pp 9233–9241 https://doi.org/10.1609/aaai.v36i8.20910
Devidze R, Radanovic G, Kamalaruban P, Singla A (2021) Explicable reward design for reinforcement learning agents. Adv Neural Inf Process Syst 34:20118–20131
Coumans E, Bai Y (2016–2021) PyBullet, a Python module for physics simulation for games, robotics and machine learning. http://pybullet.org
Brockman G, Cheung V, Pettersson L, Schneider J, Schulman J, Tang J, Zaremba W (2016) Openai gym. arXiv preprint arXiv:1606.01540
Koenig N (2004) Howard A (2004) Design and use paradigms for gazebo, an open-source multi-robot simulator. IEEE/RSJ International conference on intelligent robots and systems (IROS) vol 3, pp 2149–2154
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Shahid, M., Khan, S.N., Iqbal, K.F. et al. Dynamic Goal Tracking for Differential Drive Robot Using Deep Reinforcement Learning. Neural Process Lett 55, 11559–11576 (2023). https://doi.org/10.1007/s11063-023-11390-2
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11063-023-11390-2