Skip to main content
SpringerLink
Log in

Learning visual path–following skills for industrial robot using deep reinforcement learning

  • Application
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

The visual path–following technology is widely used in cutting, laser welding, painting, gluing, and other fields, which is a crucial content of robotics studies. As an important algorithm of artificial intelligence (AI), reinforcement learning provides a new insight for robots to learn path-following skills which has the ability of machine vision and decision making. In order to build a robotic agent with path-following skills, this paper proposes a visual path–following algorithm based on artificial intelligence deep reinforcement learning double deep Q-network (DDQN). The proposed approach allows the robot to learn path-following skill by itself, using a visual sensor in the Robot Operating System (ROS) simulation environment. The robot can learn paths with different textures, colors, and shapes, which enhances the flexibility for different industrial robot application scenarios. Skills acquired in simulation can be directly translated to the real world. In order to verify the performance of the path-following skill, a path randomly hand-drawn on the workpiece is tested by the six-joint robot Universal Robots 5 (UR5). The simulation and real experiment results demonstrate that robots can efficiently and accurately perform path following autonomously using visual information without the parameters of the path and without programming the path in advance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Ghobakhloo M (2022) Industry4.0, digitization, and opportunities for sustainability. J Clean Prod 252:119869

  2. Duan YQ, Edwards JS, Dwivedi YK (2019) Artificial intelligence for decision making in the era of Big Data – evolution, challenges and research agenda. Int J Inform Manage 48:63–71

    Article  Google Scholar 

  3. Pérez L, Rodríguez Í, Rodríguez N, Usamentiaga R, García DF (2016) Robot guidance using machine vision techniques in industrial environments: a comparative review. Sensors 16(335)

  4. Wang ZG, Wang HT, She Q, Shi XS, Zhang YM (2020) Robot4.0: continual learning and spatial-temporal intelligence through edge. J Comp Res Dev 57(9)1854–1863

  5. Mnih V, Kavukcuoglu K, Silver D, Antonoglou I, Wierstra D, Riedmiller M (2013) Playing Atari with deep reinforcement learning. Comput Sci

  6. Neto P, Mendes N, Ara´ujo R, Pires JN, Moreira AP (2012) High-level robot programming based on CAD: dealing with unpredictable environments. Ind Robot Int J Robot Res Appl 39(3):294–303

    Article  Google Scholar 

  7. Polden J, Pan Z, Larkin N, Van Duin S, Norrish J (2011) Offline programming for a complex welding system using DELMIA automation. In: Chen SB, Fang G (eds) Robotic welding, intelligence and automation. Springer, Berlin, pp 341–349

    Chapter  Google Scholar 

  8. Mnih V, Kavukcuoglu K, Silver D (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533

    Article  Google Scholar 

  9. Pahič R, Lončarević Z, Gams A, Ude A (2021) Robots kill learning in latent space of a deep autoencoder neural network. Robot Auton Syst 135:103690

    Article  Google Scholar 

  10. Bedaka AK, Vidal J, Lin CY (2019) Automatic robot path integration using three-dimensional vision and offline programming. Int J Adv Manuf Technol 102:1935–1950

    Article  Google Scholar 

  11. Deng D, Polden JW, Dong JF, Tao PY (2018) Sensor guided robot path generation for surface repair tasks on a large-scale buoyancy module. Ieee-Asme T Mech 23:636–645

    Article  Google Scholar 

  12. Tian YX, Liu HF, Li L, Wang WB, Feng JC, Xi FF, Yuan GJ (2020) Robust identification of weld seam based on region of interest operation. Adv Manuf 8:473–485

    Article  Google Scholar 

  13. Shah HNM, Sulaiman M, Shukor AK, Kamis Z (2018) Butt welding joints recognition and location identification by using local thresholding. Robot Cim-Int Manuf 51:181–188

    Article  Google Scholar 

  14. Yang L, liu YH, Peng JZ, Liang ZZ, (2020) A novel system for offline 3D seam extraction and path planning based on point cloud segmentation for arc welding robot. Robot Cim-Int Manuf 64:101929

    Article  Google Scholar 

  15. Zou YB, Wei XZ, Chen JX (2020) Conditional generative adversarial network-based training image inpainting for laser vision seam tracking. Opt Laser Eng 134:10614

    Article  Google Scholar 

  16. Zhou CM, Huang BD, Fränti P (2022) A review of motion planning algorithms for intelligent robots. J Intell Manuf 33:387–424

    Article  Google Scholar 

  17. Liu NJ, Tao L, Cai YH, Wang S (2019) A review of robot manipulation skills learning methods. Acta Automatica Sinica 45(3):458–470

    MATH  Google Scholar 

  18. Moravčík M, Schmid M, Burch N, Lisý V, Morrill D, Bard N, Davis T, Waugh K, Johanson M, Bowling M (2017) DeepStack: expert-level artifcial intelligence in heads-up no-limit poker. Science 356:508–513

    Article  MathSciNet  Google Scholar 

  19. Silver D, Hubert T, Schrittwieser J, Antonoglou I, Lai M, Guez A, Lanctot M, Sifre L, Kumaran D, Graepel T, Lillicrap T, Simonyan K, Hassabis D (2017) Mastering chess and shogi by self-play with a general reinforcement learning algorithm

  20. Jaderberg M, Czarnecki WM, Dunning I, Marris L, Lever G, Castañeda AG, Beattie C, Rabinowitz NC, Morcos AS, Ruderman A, Sonnerat N, Green T, Deason L, Leibo JZ, Silver D, Hassabis D, Kavukcuoglu K, Graepel T (2019) Human-level performance in 3D multiplayer games with population-based reinforcement learning. Science 364:859–865

    Article  MathSciNet  Google Scholar 

  21. Taitler A, Shimkin N (2017) Learning control for air hockey striking using deep reinforcement learning. Int Conf Cont Artif Intel Robot Optim IEEE

  22. Zeng A, Song S, Welker S (2018) Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. IEEE/RSJ Int Conf Intel Robot Syst (IROS) 4238–4245

  23. Hundt A, Killeen B, Greene N, Wu HT (2020) “Good robot!”: efficient reinforcement learning for multi-step visual tasks with sim to real transfer. Ieee Robot Autom Let 5(4):6724–6731

    Article  Google Scholar 

  24. Guo xw, Peng GZ, Meng YY (2021) A modified Q‑learning algorithm for robot path planning in a digital twin assembly system. Int J Adv Manuf Technol

  25. Li FM, Jiang Q, Zhang SS, Wei M, Song R (2019) Robot skill acquisition in assembly process using deep reinforcement learning. Neurocomputing 345:92–102

    Article  Google Scholar 

  26. Wen SH, Zhao YF, Yuan X, Wang ZT, Zhang D, Manfredi LG (2020) Path planning for active SLAM based on deep reinforcement learning under unknown environments. Intel Serv Robot 13:262–272

    Article  Google Scholar 

  27. Zhang T, Xiao M, Zou YB, Xiao JD (2020) Robotic constant-force grinding control with a press-and-release model and model-based reinforcement learning. Int J Adv Manuf Technol 106:589–602

    Article  Google Scholar 

  28. Meyes R, Tercan H, Roggendorf S, Thiele T, Büscher C, Obdenbusch M, Brecher C, Jeschke S, Meisen T (2017) Motion planning for industrial robots using reinforcement learning. The 50th CIRP Conf Manufac Syst

  29. Hasselt HV, Guez A, Silver D (2016) Deep reinforcement learning with double Q-learning. Proceed 13th AAAI Conf Artif Intel 2094–2100

  30. Schaul T, Quan J, Antonoglou I (2016) Prioritized experience replay. Proceed 4th Int Conf Learn Represent 322–355

  31. Shelhamer E, Long J, Darrell T (2017) Fully convolutional networks for semantic segmentation. Ieee T Pattern Anal 39(4):640–651

    Article  Google Scholar 

  32. Le N, Rathour VS, Yamazaki K, Luu K, Savvides M (2021) Deep reinforcement learning in computer vision: a comprehensive survey. Artif Intell Rev

  33. Quigley M, Gerkey B, Conley K, Faust J (2009) ROS: an open-source Robot Operating System. ICRA workshop on open source software

Download references

Funding

This research is supported by the Key Laboratory Open Fund in Autonomous Region (2020520002) and the Key Research and Development Program in Autonomous Region (202003129).

Author information

Authors and Affiliations

  1. School of Mechanical Engineering, Xinjiang University, Urumqi, 830017, China

    Guoliang Liu, Wenlei Sun, Wenxian Xie & Yangyang Xu

Authors
  1. Guoliang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wenlei Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wenxian Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yangyang Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Guoliang Liu: investigation, conceptualization, code, software, experiment, data curation, and writing of the manuscript. Wenlei Sun: funding acquisition, conceptualization, project administration. Wenxian Xie: software, code, methodology, experiment. Yangyang Xu: ROS platform, system, code, experiment, software.

Corresponding author

Correspondence to Wenlei Sun.

Ethics declarations

Ethics approval

This manuscript has not been published elsewhere, in whole or in part. All authors have made significant contributions and agree with the content of the manuscript.

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, G., Sun, W., Xie, W. et al. Learning visual path–following skills for industrial robot using deep reinforcement learning. Int J Adv Manuf Technol 122, 1099–1111 (2022). https://doi.org/10.1007/s00170-022-09800-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-022-09800-1

Keywords

Search

Navigation