Abstract
To improve the efficiency of deep reinforcement learning (DRL)-based methods for robotic path planning in the unstructured environment with obstacles, we propose a Guided Deep Reinforcement Learning (GDRL) for path planning of robotic manipulators. Firstly, we introduce guided path planning to accelerate approaching process. Secondly, we design a brand-new dense reward function in DRL-based path planning. To further improve learning efficiency, the DRL agent is only trained for collision avoidance, rather than for the whole path planning process. Many useless explorations in RL process can be eliminated with these three ideas. In order to evaluate the proposal, a Franka Emika robot with 7 joints has been considered in simulator V-Rep. The simulation results show the effectiveness of the proposed GDRL method. Compared to the pure DRL method, the GDRL method has much fewer training episodes, and converges 4× faster.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
LaValle, S.M.: Planning Algorithms. Cambridge University Press, Cambridge (2006)
Jia, Q., Chen, G., et al.: Path planning for space manipulator to avoid obstacle based on A* algorithm. J. Mech. Eng. 46(13), 109–115 (2010)
Li, H., Wang, Z., Ou, Y.: Obstacle avoidance of manipulators based on improved artificial potential field method. In: 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO), Dali, China, pp. 564–569. IEEE (2019)
Al-Hmouz, R., Gulrez, T., Al-Jumaily, A.: Probabilistic road maps with obstacle avoidance in cluttered dynamic environment. In: Proceedings of IEEE International Conference on Intelligent Sensors, Sensor Networks and Information Processing Conference (ISSNIP), Melbourne, AUS, pp. 241–245. IEEE (2004)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, MA (2018)
Katyal, K., Wang, I., Burlina, P.: Leveraging deep reinforcement learning for reaching robotic tasks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, pp. 490–491. IEEE (2017)
Kamali, K., Bonev, I.A., Desrosiers, C.: Real-time motion planning for robotic teleoperation using dynamic-goal deep reinforcement learning. In: 2020 17th Conference on Computer and Robot Vision (CRV), Ottawa, Canada, pp. 182–189. IEEE (2020)
Li, Z., Ma, H., et al.: Motion planning of six-dof arm robot based on improved DDPG algorithm. In: 2020 39th Chinese Control Conference (CCC), Shenyang, China, pp. 3954–3959. IEEE (2020)
Mnih, V., Kavukcuoglu, K., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)
Gu, S., Lillicrap, T.P., et al.: Continuous deep q-learning with model-based acceleration. In: Proceedings of the 33rd International Conference on Machine Learning, New York, USA, pp. 2829–2838 (2016)
Lillicrap, T.P., Hunt, J.J., et al.: Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015)
Mnih, V., Badia, A.P., et al.: Asynchronous methods for deep reinforcement learning. In: Proceedings of the 33rd International Conference on Machine Learning, New York, USA, pp. 1928–1937 (2016)
Schulman, J., Wolski, F., et al.: Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347 (2017)
Schulman, J., Levine, S., et al.: Trust region policy optimization. In: Proceedings of the 31st International Conference on Machine Learning, Lille, France, pp. 1889–1897 (2015)
Fujimoto, S., Hoof, H.V., et al.: Addressing function approximation error in actor-critic methods. arXiv preprint arXiv:1802.09477 (2018)
Haarnoja, T., Zhou, A., et al.: Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. arXiv preprint arXiv:1801.01290 (2018)
Sangiovanni, B., Rendiniello, A., et al.: Deep reinforcement learning for collision avoidance of robotic manipulators. In: 2018 European Control Conference (ECC), Limassol, Cyprus, pp. 2063–2068. IEEE (2018)
Xie, J., Shao, Z., et al.: Deep reinforcement learning with optimized reward functions for robotic trajectory planning. IEEE Access 2019(7), 105669–105679 (2019)
Zeng, R., Liu, M., et al.: Manipulator control method based on deep reinforcement learning. In: 2020 Chinese Control and Decision Conference (CCDC), Hefei, China, pp. 415–420. IEEE (2020)
James, S., Freese, M., Davison, A.J.: PyRep: Bringing V-REP to Deep Robot Learning. arXiv preprint arXiv:1906.11176 (2019)
Acknowledgement
This work was supported by Major Project of the New Generation of Artificial Intelligence (No. 2018AAA0102900).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Shen, Y., Jia, Q., Huang, Z., Wang, R., Chen, G. (2021). Guided Deep Reinforcement Learning for Path Planning of Robotic Manipulators. In: Sun, F., Liu, H., Fang, B. (eds) Cognitive Systems and Signal Processing. ICCSIP 2020. Communications in Computer and Information Science, vol 1397. Springer, Singapore. https://doi.org/10.1007/978-981-16-2336-3_27
Download citation
DOI: https://doi.org/10.1007/978-981-16-2336-3_27
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-16-2335-6
Online ISBN: 978-981-16-2336-3
eBook Packages: Computer ScienceComputer Science (R0)