Skip to main content
SpringerLink
Log in

Dynamic visual servoing of large-scale dual-arm cooperative manipulators based on photogrammetry sensor

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

The dual-arm cooperative manipulators system has more advantages in reconfigurability and flexibility compared with single manipulator manufacturing cell. However, random and time-varying Cartesian pose errors generated in dual-arm cooperative manipulators will be superimposed to increase the system synchronous error, which makes it difficult to complete the coordinate assembly task. To deal with the above-mentioned problems, in this paper, a novel cooperative mode-based dynamic cross-coupled sliding mode controller (DCSMC) scheme combined with position-based visual servoing (PBVS) is proposed to realize 3D dynamic tracking and positioning by correcting the movement of the dual-arm system in real time. Then, since the cross-coupled technology is incorporated into the proposed synchronous control scheme for the dual-arm system, both the Cartesian position tracking and synchronous errors of the end-effectors will simultaneously converge to zero. Moreover, the stability of the proposed DCSMC has been proved by using the Lyapunov theorem. Furthermore, to effectively demonstrate the control performance of the proposed synchronous control approach, a real-time experimental platform integrated with BECKHOFF industrial computer, C-track 780 photogrammetry sensor from Creaform, and two ESTUN industrial manipulators were developed for the implementation of the proposed DCSMC system. Finally, experimental results illustrate that the proposed scheme can improve the synchronous accuracy up to ± 0.13 mm for position accuracy and ± 62.6 × 10−5 rad for rotational accuracy respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Data availability

The data sets supporting the results of this article are included within the article.

Code availability

Not applicable.

References

  1. Tian W, Zeng Y, Zhou W, Liao W (2014) Calibration of robotic drilling systems with a moving rail[J]. Chin J Aeronaut 27(6):1598–1604

    Article  Google Scholar 

  2. Li B, Tian W, Zhang C, Hua F, Cui G, Li Y (2022) Positioning error compensation of an industrial robot using neural networks and experimental study[J]. Chin J Aeronaut 35(2):346–360

    Article  Google Scholar 

  3. Cui G, Li B, Tian W, Liao W, Zhao W (2022) Dynamic modeling and vibration prediction of an industrial robot in manufacturing[J]. Appl Math Model 105:114–136

    Article  MathSciNet  MATH  Google Scholar 

  4. Bourne D, Choset H, Hu H, Kantor G, Niessl C, Rubinstein Z, Simmons R, Smith S (2015) Mobile manufacturing of large structures[C]. Proc 2015 IEEE Int Conf Robot Autom (ICRA), pp 1565–1572. https://doi.org/10.1109/ICRA.2015.7139397

  5. Bøgh S, Hvilshø j M, Kristiansen M, Madsen O (2011) Identifying and evaluating suitable tasks for autonomous industrial mobile manipulators (aimm)[J]. Int J Adv Manuf Technol 61:713–726

    Article  Google Scholar 

  6. Zhou K, Ebenhofer G, Eitzinger C, Zimmermann U, Walter C, Saenz J, Castaño LP, Hernández MAF, Oriol JN (2014) Mobile manipulator is coming to aerospace manufacturing industry[C]. Proc 2014 IEEE Int Symp Robot Sensors Environ (ROSE) Proc, pp 16–18. https://doi.org/10.1109/ROSE.2014.6952990

  7. Chen J, Xie F, Liu X-J, Bi W (2021) Stiffness evaluation of an adsorption robot for large-scale structural parts processing[J]. J Mech Robot 13(4):040907. https://doi.org/10.1115/1.4050683

    Article  Google Scholar 

  8. Shu T, Gharaaty S, Xie W, Joubair A, Bonev IA (2018) Dynamic path tracking of industrial robots with high accuracy using photogrammetry sensor[J]. IEEE/ASME Trans Mechatron 23(3):1159–1170

    Article  Google Scholar 

  9. Basile F, Caccavale F, Chiacchio P, Coppola J, Curatella C (2012) Task-oriented motion planning for multi-arm robotic systems[J]. Robot Comput-Integr Manuf 28(5):569–582

    Article  Google Scholar 

  10. Everhart T (2017) Neighboring mobile robot cell with drilling and fastening[C]. SAE Techni Paper 2017-01-2094. https://doi.org/10.4271/2017-01-2094

  11. Prat P, Gueydon E (2011) Cooperative robots for full automation[C]. SAE Techni Paper 2011-01-2536. https://doi.org/10.4271/2011-01-2536

  12. Lv N, Liu J, Jia Y (2021) Coordinated control of flexible cables with human-like dual manipulators[J]. J Dyn Syst Meas Control 143(8):081006. https://doi.org/10.1115/1.4050398

    Article  Google Scholar 

  13. Zhu Q, Xie X, Li C, Xia G, Liu Q (2019) Kinematic self-calibration method for dual-manipulators based on optical axis constraint[J]. IEEE Access 7:7768–7782

    Article  Google Scholar 

  14. Feng Z, Hu G, Sun Y, Soon J (2020) An overview of collaborative robotic manipulation in multi-robot systems[J]. Annu Rev Control 49:113–127. https://doi.org/10.1016/j.arcontrol.2020.02.002

    Article  MathSciNet  Google Scholar 

  15. Brahmi A, Saad M, Gauthier G, Zhu W-H, Ghommam J (2017) Adaptive control of multiple mobile manipulators transporting a rigid object[J]. Int J Control Autom Syst 15:1779–1789. https://doi.org/10.1007/s12555-015-0116-x

    Article  Google Scholar 

  16. Kornmaneesang W, Chen S-L (2021) Mpc-based robust contouring control for a robotic machining system[J]. Asian J Control 23:1212–1224

    Article  Google Scholar 

  17. Jinjun D, Yahui G, Ming C, Xianzhong D (2019) Symmetrical adaptive variable admittance control for position/force tracking of dual-arm cooperative manipulators with unknown trajectory deviations[J]. Robot Comput-Integr Manuf 57:357–369

    Article  Google Scholar 

  18. Guo S, Nie S-L, Xi F-F, Song T (2014) Modeling and simulation of percussive impact for robotic riveting system[J]. Adv Manuf 2(4):344–352

    Article  Google Scholar 

  19. Torres F, Puente S, Díaz C (2009) Automatic cooperative disassembly robotic system: task planner to distribute tasks among robots[J]. Control Eng Pract 17(1):112–121

    Article  Google Scholar 

  20. Stadelmann L, Sandy T, Thoma A, Buchli J (2019) End-effector pose correction for versatile large-scale multi-robotic systems[J]. IEEE Robot Autom Lett 4(2):546–553

    Article  Google Scholar 

  21. Su K-H, Cheng M-Y (2008) Contouring accuracy improvement using cross-coupled control and position error compensator[J]. Int J Mach Tools Manuf 48(12):1444–1453

    Article  Google Scholar 

  22. Lin F, Chou P, Chen C, Lin Y (2012) Dsp-based cross-coupled synchronous control for dual linear motors via intelligent complementary sliding mode control[J]. IEEE Trans Industr Electron 59(2):1061–1073

    Article  Google Scholar 

  23. Ouyang PR, Dam T, Pano V (2015) Cross-coupled pid control in position domain for contour tracking[J]. Robotica 33(6):1351–1374

    Article  MATH  Google Scholar 

  24. Zhao D, Zhu Q (2014) Position synchronised control of multiple robotic manipulators based on integral sliding mode[J]. Int J Syst Sci 45:556–570

    Article  MathSciNet  MATH  Google Scholar 

  25. Dong S, Mills JK (2002) Adaptive synchronized control for coordination of multirobot assembly tasks[J]. IEEE Trans Robot Autom 18(4):498–510

    Article  Google Scholar 

  26. Dong S, Mills JK (2002) Adaptive synchronized control for coordination of two robot manipulators[C]. Proc Proc 2002 IEEE Int Conf Robot Automa (Cat No02CH37292), pp 976–981. https://doi.org/10.1109/ROBOT.2002.1013482

  27. Li S, Ghasemi A, Xie W, Gao Y (2018) An enhanced ibvs controller of a 6dof manipulator using hybrid pd-smc method[C]. Proc IECON - Annu Conf IEEE Ind Electron Soc, pp 2852–2857. https://doi.org/10.1109/IECON.2017.8216481

  28. Xie W, Li Z, Tu X, Perron C (2009) Switching control of image-based visual servoing with laser pointer in robotic manufacturing systems[J]. IEEE Trans Industr Electron 56(2):520–529

    Article  Google Scholar 

  29. Chang W-C, Cheng M-Y, Tsai H-J (2017) Image feature command generation of contour following tasks for scara robots employing image-based visual servoing—a ph-spline approach[J]. Robot Comput-Integr Manuf 44:57–66

    Article  Google Scholar 

  30. Norouzi-Ghazbi S, Mehrkish A, Fallah MMH, Janabi-Sharifi F (2021) Constrained visual predictive control of tendon-driven continuum robots[J]. Robot Auton Syst 145:103856

    Article  Google Scholar 

  31. Zhao YM, Lin Y, Xi F, Guo S, Ouyang P (2016) Switch-based sliding mode control for position-based visual servoing of robotic riveting system[J]. J Manuf Sci Eng 139(4):041010. https://doi.org/10.1115/1.4034681

    Article  Google Scholar 

  32. Peng Y-C, Jivani D, Radke RJ, Wen J (2020) Comparing position-and image-based visual servoing for robotic assembly of large structures[C]. IEEE Int Conf Autom Sci Eng, pp 1608–1613. https://doi.org/10.1109/CASE48305.2020.9217028

  33. Al-Shanoon A, Lang H (2022) Robotic manipulation based on 3-d visual servoing and deep neural networks[J]. Robot Auton Syst 152:104041

    Article  Google Scholar 

  34. Lippiello V, Cacace J, Santamaria-Navarro A, Andrade-Cetto J, Trujillo MÁ, Esteves YRR, Viguria A (2016) Hybrid visual servoing with hierarchical task composition for aerial manipulation[J]. IEEE Robot Autom Lett 1(1):259–266

    Article  Google Scholar 

  35. Gao J, An X, Proctor A, Bradley C (2017) Sliding mode adaptive neural network control for hybrid visual servoing of underwater vehicles[J]. Ocean Eng 142:666–675

    Article  Google Scholar 

  36. Du Y-C, Taryudi T, Tsai C-T, Wang M-S (2019) Eye-to-hand robotic tracking and grabbing based on binocular vision[J]. Microsyst Technol : Sensors Actuators Syst Integr 27(4):1699–1710

    Article  Google Scholar 

  37. Wang H, Guo D, Xu H, Chen W, Liu T, Leang KK (2017) Eye-in-hand tracking control of a free-floating space manipulator[J]. IEEE Trans Aerosp Electron Syst 53(4):1855–1865

    Article  Google Scholar 

  38. Li P, Shu T, Xie W-F, Tian W (2021) Dynamic visual servoing of a 6-rss parallel robot based on optical cmm[J]. J Intell Rob Syst 102(2):40

    Article  Google Scholar 

  39. Muñoz-Benavent P, Gracia L, Solanes JE, Esparza A, Tornero J (2018) Sliding mode control for robust and smooth reference tracking in robot visual servoing[J]. Int J Robust Nonlinear Control 28(5):1728–1756

    Article  MathSciNet  MATH  Google Scholar 

Download references

Funding

This work has been supported by the National Key R & D Program of China under Grant No. 2020YFB1710300, and Natural Science Foundation of China under Grant Nos. 52075256, 52005254, 52205530, and 62003346, and Natural Science Foundation of Jiangsu Province under Grant No. BK20210299.

Author information

Authors and Affiliations

  1. College of Mechanical and Electrical Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing, 210016, China

    Quan Bai, Pengcheng Li, Wei Tian, Jianxin Shen & Bo Li

  2. Beijing Spacecrafts, China Academy of Space Technology, Beijing, 100094, China

    Ke Wen

Authors
  1. Quan Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pengcheng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wei Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jianxin Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bo Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ke Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Quan Bai: test design, test execution, software, data analysis, writing—original draft preparation. Pengcheng Li: test design, investigation, reviewing and editing, project administration. Wei Tian: test design, data analysis, writing and editing. Jianxin Shen: validation, project administration, writing—reviewing and editing. Bo Li: investigation, supervision, reviewing and editing. Ke Wen: writing—reviewing and editing.

Corresponding authors

Correspondence to Pengcheng Li or Wei Tian.

Ethics declarations

Ethics approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

We confirm that this work is original and has not been published elsewhere, nor is it currently under consideration for publication elsewhere. All the authors listed have agreed to publish the manuscript that is enclosed.

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Highlights

1. Modeling the error kinematic with the consideration of a lumped uncertainty of dual 6-DOF industrial manipulators.

2. The visual servoing control scheme based on the sliding mode control method and cross-coupling technology is conducted.

3. Experimental validation shows high-performance position tracking response and synchronous control.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bai, Q., Li, P., Tian, W. et al. Dynamic visual servoing of large-scale dual-arm cooperative manipulators based on photogrammetry sensor. Int J Adv Manuf Technol 126, 4037–4054 (2023). https://doi.org/10.1007/s00170-023-11337-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-023-11337-w

Keywords

Search

Navigation