Abstract
During the movement of a robotic arm, collisions can easily occur if the arm directly grasps at multiple tightly stacked objects, thereby leading to grasp failures or machine damage. Grasp success can be improved through the rearrangement or movement of objects to clear space for grasping. This paper presents a high-performance deep Q-learning framework that can help robotic arms to learn synchronized push and grasp tasks. In this framework, a grasp quality network is used for precisely identifying stable grasp positions on objects to expedite model convergence and solve the problem of sparse rewards caused during training because of grasp failures. Furthermore, a novel reward function is proposed for effectively evaluating whether a pushing action is effective. The proposed framework achieved grasp success rates of 92% and 89% in simulations and real-world experiments, respectively. Furthermore, only 200 training steps were required to achieve a grasp success rate of 80%, which indicates the suitability of the proposed framework for rapid deployment in industrial settings.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Data Availability
The data that support the findings of this study are not available due to their necessity for future experiments.
References
Kong, L., He, W., Yang, W., Li, Q., Kaynak, O.: Fuzzy approximation-based finite-time control for a robot with actuator saturation under time-varying constraints of work space. IEEE Trans. Cybern. 51(10), 4873–4884 (2021). https://doi.org/10.1109/TCYB.2020.2998837
Kong, L., He, W., Yang, C., Li, Z., Sun, C.: adaptive fuzzy control for coordinated multiple robots with constraint using impedance learning. IEEE Trans. Cybern. 49(8), 3052–3063 (2019). https://doi.org/10.1109/TCYB.2018.2838573
Kong, L., He, W., Dong, Y., Cheng, L., Yang, C., Li, Z.: Asymmetric bounded neural control for an uncertain robot by state feedback and output feedback. IEEE Trans. Syst. Man, Cybern. Syst. 51(3), 1735–1746 (2021). https://doi.org/10.1109/TSMC.2019.2901277
Miller, A.T., Knoop, S., Christensen, H.I., Allen, P.K.: Automatic grasp planning using shape primitives. In: 2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422), vol. 2, pp. 1824–1829 (2003). https://doi.org/10.1109/ROBOT.2003.1241860
Bohg, J., Morales, A., Asfour, T., Kragic, D.: Data-driven grasp synthesis—a survey. IEEE Trans. Rob. 30(2), 289–309 (2014). https://doi.org/10.1109/TRO.2013.2289018
Tian, H., Song, K., Li, S., Ma, S., Xu, J., Yan, Y.: Data-driven robotic visual grasping detection for unknown objects: a problem-oriented review. Exp. Syst. Appl. 211, 118624 (2023). https://doi.org/10.1016/j.eswa.2022.118624
He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2980-2988 (2017). https://doi.org/10.1109/ICCV.2017.322
Cao, Z., Simon, T., Wei, S.E., Sheikh, Y.: Realtime multi-person 2D pose estimation using part affinity fields. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1302-1310 (2017). https://doi.org/10.1109/CVPR.2017.143
Mnih, V. et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015). https://doi.org/10.1038/nature14236
Zeng, A., Song, S., Welker, S., Lee, J., Rodriguez, A., Funkhouser, T.: Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4238-4245 (2018). https://doi.org/10.1109/IROS.2018.8593986
Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3431-3440 (2015). https://doi.org/10.1109/CVPR.2015.7298965
Berscheid, L., Meißner, P., Kröger, T.: Robot Learning of Shifting Objects for Grasping in Cluttered Environments. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 612-618 (2019). https://doi.org/10.1109/IROS40897.2019.8968042
Yang, Y., Ni, Z., Gao, M., Zhang, J., Tao, D.: Collaborative pushing and grasping of tightly stacked objects via deep reinforcement learning. IEEE/CAA J. Autom. Sin. 9(1), 135–145 (2022). https://doi.org/10.1109/JAS.2021.1004255
Xu, K., Yu, H., Lai, Q., Wang, Y., Xiong, R.: Efficient learning of goal-oriented push-grasping synergy in clutter. IEEE Robot. Autom. Lett. 6(4), 6337–6344 (2021). https://doi.org/10.1109/LRA.2021.3092640
Mahler, J. et al.:Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics (2017). https://doi.org/10.48550/arXiv.1703.09312
Sahbani, A., El-Khoury, S., Bidaud, P.: An overview of 3D object grasp synthesis algorithms. Robot. Auton. Syst. 60(3), 326–336 (2012). https://doi.org/10.1016/j.robot.2011.07.016
Chu, F.J., Xu, R., Vela, P.A.: Real-world multiobject, multigrasp detection. IEEE Rob. Autom. Lett. 3(4), 3355–3362 (2018). https://doi.org/10.1109/LRA.2018.2852777
Morrison, D., Corke, P., Leitner, J.: Closing the loop for robotic grasping: A real-time, generative grasp synthesis approach (2018). https://doi.org/10.48550/arXiv.1804.05172
Sundermeyer, M., Mousavian, A., Triebel, R., Fox, R.: Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 13438-13444 (2021). https://doi.org/10.1109/ICRA48506.2021.9561877
Ghadirzadeh, A., Maki, A., Kragic, D., Björkman, M.: Deep predictive policy training using reinforcement learning. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2351-2358 (2017). https://doi.org/10.1109/IROS.2017.8206046
Quillen, D., Jang, E., Nachum, O., Finn, C., Ibarz, J., Levine, S.: Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methods. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 6284-6291 (2018). https://doi.org/10.1109/ICRA.2018.8461039
Yen-Chen, L., Zeng, A., Song, S., Isola, P., Lin, T.Y.: Learning to See before Learning to Act: Visual Pre-training for Manipulation. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 7286-7293 (2020). https://doi.org/10.1109/ICRA40945.2020.9197331
Kalashnikov, D. et al.: Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation (2018). https://doi.org/10.48550/arXiv.1806.10293
Yang, Y., Liang, H., Choi, C.: A deep learning approach to grasping the invisible. IEEE Robot. Autom. Lett. 5(2), 2232–2239 (2020). https://doi.org/10.1109/LRA.2020.2970622
Li, E., Feng, H., Zhang, S., Fu, Y.: Learning target-oriented push-grasping synergy in clutter with action space decoupling. IEEE Robot. Autom. Lett. 7(4), 11966–11973 (2022). https://doi.org/10.1109/LRA.2022.3204822
Florence, P., et al.: Implicit behavioral cloning. In: Conference on Robot Learning, pp. 158–168: PMLR (2022). https://doi.org/10.48550/arXiv.2109.00137
Zeng, A., et al.: Transporter networks: Rearranging the visual world for robotic manipulation. In: Conference on Robot Learning, pp. 726–747 : PMLR (2021). https://doi.org/10.48550/arXiv.2010.14406
Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. Presented at the Proceedings of the 27th International Conference on International Conference on Machine Learning, Haifa, Israel (2010)
Huang, B., Han, S.D., Boularias, A., Yu, J.: DIPN: Deep Interaction Prediction Network with Application to Clutter Removal. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 4694-4701 (2021). https://doi.org/10.1109/ICRA48506.2021.9561073
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90
Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2117–2125 (2017). https://doi.org/10.1109/CVPR.2017.106
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708 (2017). https://doi.org/10.1109/CVPR.2017.243
Schaul, T., Quan, J., Antonoglou, I., Silver, D.: Prioritized experience replay. (2015). https://doi.org/10.48550/arXiv.1511.05952
Rohmer, E., Singh, S. P. N., Freese, M.: V-REP: A versatile and scalable robot simulation framework. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1321-1326 (2013). https://doi.org/10.1109/IROS.2013.6696520
Hundt, A., et al.: “Good Robot!”: efficient reinforcement learning for multi-step visual tasks with sim to real transfer. IEEE Robot. Autom. Lett. 5(4), 6724–6731 (2020). https://doi.org/10.1109/LRA.2020.3015448
Funding
The authors declare that no funds, grants, or other support were received during the preparation of this manuscript.
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception, design and methodology. The experiment were performed by Yu-Hsiang Shao. The first draft of the manuscript was written by Chih-Yung Huang and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethics Approval
Not applicable.
Consent to Participate
Not applicable.
Consent to Publish
Not applicable.
Competing Interests
The authors have no relevant financial or non-financial interests to disclose.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Huang, CY., Shao, YH. Integration of Deep Q-Learning with a Grasp Quality Network for Robot Grasping in Cluttered Environments. J Intell Robot Syst 110, 97 (2024). https://doi.org/10.1007/s10846-024-02127-x
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10846-024-02127-x