Skip to main content
SpringerLink
Log in

Utilizing deep reinforcement learning for tactile-based autonomous capture of non-cooperative objects in space

  • Original Paper
  • Published:
Aerospace Systems Aims and scope Submit manuscript

Abstract

The focus of this research is the creation of a deep reinforcement learning approach to tackle the challenging task of robotic gripping through tactile sensor data feedback. Leveraging deep reinforcement learning, we have sidestepped the necessity to design features manually, which simplifies the issue and allows the robot to acquire gripping strategies via trial-and-error learning. Our technique utilizes an off-policy reinforcement learning model, integrating deep deterministic policy gradient structure and twin delayed attributes to facilitate maximum precision in gripping floating items. We have formulated a comprehensive reward function to provide the agent with precise, insightful feedback to facilitate the learning of the gripping task. The training of our model was executed solely in a simulated environment using the PyBullet framework and did not require demonstrations or pre-existing knowledge of the task. We examined a gripping task with a 3-finger Robotiq gripper for a case study, where the gripper had to approach a floating object, pursue it, and eventually grip it. This training methodology in a simulated setting allowed us to experiment with various scenarios and conditions, thereby enabling the agent to develop a resilient and adaptable grip policy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Algorithm 1
Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data availability

Data is not available publicly for this paper.

References

  1. Shan M, Guo J, Gill E (2016) Review and comparison of active space debris capturing and removal methods. Prog Aerosp Sci 80:18–32. https://doi.org/10.1016/j.paerosci.2015.11.001

    Article  Google Scholar 

  2. Zhao P, Liu J, Wu C (2020) Survey on research and development of on-orbit active debris removal methods. Sci China Technol Sci 63(11):2188–2210. https://doi.org/10.1007/s11431-020-1661-7

    Article  Google Scholar 

  3. Macauley MK (2015) The economics of space debris: Estimating the costs and benefits of debris mitigation. Acta Astronaut 115:160–164. https://doi.org/10.1016/j.actaastro.2015.05.006

    Article  Google Scholar 

  4. Schaub H, Jasper LEZ, Anderson PV, McKnight DS (2015) Cost and risk assessment for spacecraft operation decisions caused by the space debris environment. Acta Astronaut 113:66–79. https://doi.org/10.1016/j.actaastro.2015.03.028

    Article  Google Scholar 

  5. Rybus T (2018) Obstacle avoidance in space robotics: Review of major challenges and proposed solutions. Prog Aerosp Sci 101:31–48. https://doi.org/10.1016/j.paerosci.2018.07.001

    Article  Google Scholar 

  6. Ledkov A, Aslanov V (2022) Review of contact and contactless active space debris removal approaches. Prog Aerosp Sci 134:100858. https://doi.org/10.1016/j.paerosci.2022.100858

    Article  Google Scholar 

  7. Matney M et al (2019) The NASA orbital debris engineering model 3.1: development, verification, and validation. In: International orbital debris conference (IOC)

  8. Papadopoulos E, Aghili F, Ma O, Lampariello R (2021) Robotic manipulation and capture in space: a survey. Front Robot AI. https://doi.org/10.3389/frobt.2021.686723

    Article  Google Scholar 

  9. Sun Y, Falco J, Roa MA, Calli B (2022) Research challenges and progress in robotic grasping and manipulation competitions. IEEE Robot Autom Lett 7(2):874–881. https://doi.org/10.1109/LRA.2021.3129134

    Article  Google Scholar 

  10. Mnih V et al (2013) Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602

  11. Mnih V et al (2015) Human-level control through deep reinforcement learning. Nature 518:7540. https://doi.org/10.1038/nature14236

    Article  Google Scholar 

  12. Lample G, Chaplot DS (2017) Playing FPS games with deep reinforcement learning. In: Proceedings of the AAAI conference on artificial intelligence. https://doi.org/10.1609/aaai.v31i1.10827

  13. Duan Y, Chen X, Houthooft R, Schulman J, Abbeel P (2016) Benchmarking deep reinforcement learning for continuous control. In: Proceedings of the 33rd international conference on machine learning, PMLR, June 2016, pp. 1329–1338. Available https://proceedings.mlr.press/v48/duan16.html. Accessed July 17, 2023

  14. Guo D, Sun F, Fang B, Yang C, Xi N (2017) Robotic grasping using visual and tactile sensing. Inf Sci 417:274–286. https://doi.org/10.1016/j.ins.2017.07.017

    Article  Google Scholar 

  15. Melnik A, Lach L, Plappert M, Korthals T, Haschke R, Ritter H (2019) Tactile sensing and deep reinforcement learning for in-hand manipulation tasks. In: IROS workshop on autonomous object manipulation

  16. Merzić H, Bogdanović M, Kappler D, Righetti L, Bohg J (2019) Leveraging contact forces for learning to grasp. In: 2019 international conference on robotics and automation (ICRA), May 2019, pp 3615–3621. https://doi.org/10.1109/ICRA.2019.8793733.

  17. Liu H et al (2022) Multi-fingered tactile serving for grasping adjustment under partial observation. In: 2022 IEEE/RSJ international conference on intelligent robots and systems (IROS), Oct. 2022, pp 7781–7788. https://doi.org/10.1109/IROS47612.2022.9981464

  18. Koenig A, Liu Z, Janson L, Howe R (2021) Tactile grasp refinement using deep reinforcement learning and analytic grasp stability metrics. arXiv:2109.11234 [cs, eess], Sep. 2021. Available http://arxiv.org/abs/2109.11234. Accessed: Dec. 15, 2021

  19. Tai L, Zhang J, Liu M, Boedecker J, Burgard W (2018) A survey of deep network solutions for learning control in robotics: from reinforcement to imitation. arXiv, Apr. 8, 2018. https://doi.org/10.48550/arXiv.1612.07139.

  20. Benning M, Celledoni E, Ehrhardt MJ, Owren B, Schönlieb C-B (2019) Deep learning as optimal control problems: models and numerical methods. arXiv, Sep. 30, 2019. https://doi.org/10.48550/arXiv.1904.05657.

  21. Lenz I, Lee H, Saxena A (2015) Deep learning for detecting robotic grasps. Int J Robot Res 34(4–5):705–724. https://doi.org/10.1177/0278364914549607

    Article  Google Scholar 

  22. Saxena A, Driemeyer J, Ng AY (2008) Robotic grasping of novel objects using vision. Int J Robot Res 27(2):157–173. https://doi.org/10.1177/0278364907087172

    Article  Google Scholar 

  23. Kumar V, Hermans T, Fox D, Birchfield S, Tremblay J (2019) Contextual reinforcement learning of visuo-tactile multi-fingered grasping policies. arXiv preprint, arXiv:1911.09233

  24. Sutton RS, Barto AG (2018) reinforcement learning: an introduction. MIT Press, Cambridge

    MATH  Google Scholar 

  25. Buşoniu L, de Bruin T, Tolić D, Kober J, Palunko I (2018) Reinforcement learning for control: performance, stability, and deep approximators. Annu Rev Control 46:8–28. https://doi.org/10.1016/j.arcontrol.2018.09.005

    Article  MathSciNet  Google Scholar 

  26. Li Y (2017) Deep reinforcement learning: an overview. arXiv preprint, arXiv:1701.07274

  27. Arulkumaran K, Deisenroth MP, Brundage M, Bharath AA (2017) Deep reinforcement learning: a brief survey. IEEE Signal Process Mag 34(6):26–38. https://doi.org/10.1109/MSP.2017.2743240

    Article  Google Scholar 

  28. Kleeberger K, Bormann R, Kraus W, Huber MF (2020) A survey on learning-based robotic grasping. Curr Robot Rep 1(4):239–249. https://doi.org/10.1007/s43154-020-00021-6

    Article  Google Scholar 

  29. Fujimoto S, Hoof H, Meger D (2022) Addressing function approximation error in actor-critic methods. In: Proceedings of the 35th international conference on machine learning, PMLR, July 2018, pp 1587–1596. Available https://proceedings.mlr.press/v80/fujimoto18a.html. Accessed Dec. 15, 2022

  30. Lillicrap TP et al (2019) Continuous control with deep reinforcement learning. arXiv, Jul. 05, 2019. https://doi.org/10.48550/arXiv.1509.02971

  31. Dankwa S, Zheng W (2020) Twin-delayed DDPG: a deep reinforcement learning technique to model a continuous movement of an intelligent robot agent. In: Proceedings of the 3rd international conference on vision, image and signal processing, in ICVISP 2019, May 2020. Association for Computing Machinery, New York, NY, USA, pp 1–5. https://doi.org/10.1145/3387168.3387199.

  32. Raffin A, Hill A, Gleave A, Kanervisto A, Ernestus M, Dormann N (2021) Stable-baselines3: reliable reinforcement learning implementations. J Mach Learn Res 22:1–8

    MATH  Google Scholar 

  33. E. Coumans and Y. Bai. (2017). Pybullet, a Python Module for Physics Simulation in Robotics, Games and Machine Learning. [Online]. Available: https://pybullet.org

  34. Brockman G et al (2016) Openai gym. arXiv preprint arXiv:1606.01540

Download references

Acknowledgements

This work is funded by the Discovery Grant (RGPIN-2018-05991) and CREATE Grant (555425-2021) of the Natural Sciences and Engineering Research Council of Canada, and FAST grant (19FAYORA14) of the Canadian Space Agency.

Funding

Natural Sciences and Engineering Research Council of Canada, Discovery Grant/RGPIN-2018-05991, Zheng Hong Zhu, CREATE Grant/555425-2021, Zheng Hong Zhu, Canadian Space Agency, FAST grant/19FAYORA14, Zheng Hong Zhu.

Author information

Authors and Affiliations

  1. Department of Mechanical Engineering, York University, 4700 Keele Street, Toronto, Canada

    Bahador Beigomi & Zheng H. Zhu

Authors
  1. Bahador Beigomi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zheng H. Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All the authors contributed to the study conception and design. Material preparation, data collection, and analysis were performed by BB and ZHZ. The first draft of the manuscript was written by BB, and all the authors commented on previous versions of the manuscript. All the authors read and approved the final manuscript.

Corresponding author

Correspondence to Zheng H. Zhu.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Beigomi, B., Zhu, Z.H. Utilizing deep reinforcement learning for tactile-based autonomous capture of non-cooperative objects in space. AS (2023). https://doi.org/10.1007/s42401-023-00254-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42401-023-00254-1

Keywords

Search

Navigation