Skip to main content
SpringerLink
Log in

HRC of intelligent assembly system based on multimodal gesture control

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

As a natural expression of the human body, gestures are widely used in robot control. However, most gesture control methods rely only on single-modal feature representation, which has certain instability. To this end, this paper proposes a human-robot collaboration (HRC) framework for intelligent assembly systems with multimodal gesture control. The framework aims to perform gesture recognition on RGB data and RGB-D data through convolutional neural networks (CNN) and combine gesture data from both modalities for application in robot control, enabling data fusion and feature sharing. Perform asynchronous integration of image acquisition and gesture detection with spatially and time-aligned RGB frames and RGB-D frames to ensure real-time detection of gestures, and speed and separation monitoring to ensure the safety of the collaboration process. To verify the effectiveness of this framework, it was applied to the assembly scene of the integrally shrouded blade-rotor system. Experiments show that the HRC of the intelligent assembly system with multimodal gesture control can better realize the human-robot collaborative assembly with the integrally shrouded blade and improve the intelligence of the assembly process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data Availability

All data generated or analyzed during this study are included in the present article.

References

  1. Gervasi R, Mastrogiacomo L, Franceschini F (2020) A conceptual framework to evaluate human-robot collaboration. Int J Adv Manuf Technol 108:841–865. https://doi.org/10.1007/s00170-020-05363-1

    Article  Google Scholar 

  2. Liu H, Wang L (2018) Gesture recognition for human-robot collaboration: a review. Int J Ind Ergon 68:355–367. https://doi.org/10.1016/j.ergon.2017.02.004

    Article  Google Scholar 

  3. ISO/TS 15066 (2016) Robots and robotic devices - safety requirements for collaborative industrial robot systems and the work environment: Collaborative robots. ISO/TS; (2016)

    Google Scholar 

  4. Maitre J, Rendu C, Bouchard K, Bouchard B, Gaboury S (2019) Basic daily activity recognition with a data glove. Procedia Comput Sci 151:108–115. https://doi.org/10.1016/j.procs.2019.04.018

    Article  Google Scholar 

  5. Kakoty NM, Sharma MD (2018) Recognition of sign language alphabets and numbers based on hand kinematics using a data glove. Procedia Comput Sci 133:55–62. https://doi.org/10.1016/j.procs.2018.07.008

    Article  Google Scholar 

  6. Shrenika S, Bala MM (2020) Sign language recognition using template matching technique. In: 2020 international conference on computer science, engineering and applications (ICCSEA), pp 1–5. https://doi.org/10.1109/iccsea49143.2020.9132899

    Chapter  Google Scholar 

  7. Avola D, Cinque L, Fagioli A, Foresti GL, Fragomeni A, Pannone D (2022) 3D hand pose and shape estimation from RGB images for key point-based hand gesture recognition. Pattern Recogn 129:108762. https://doi.org/10.1016/j.patcog.2022.108762

    Article  Google Scholar 

  8. Sharma S, Singh S (2021) Vision-based hand gesture recognition using deep learning for the interpretation of sign language. Expert Syst Appl 182:115657. https://doi.org/10.1016/j.eswa.2021.115657

    Article  Google Scholar 

  9. Zhang Y, Shi L, Wu Y, Cheng K, Cheng J, Lu H (2020) Gesture recognition based on deep deformable 3D convolutional neural networks. Pattern Recogn 107:107416. https://doi.org/10.1016/j.patcog.2020.107416

    Article  Google Scholar 

  10. Neto P, Simão M, Mendes N, Safeea M (2019) Gesture-based human-robot interaction for human assistance in manufacturing. Int J Adv Manuf Technol 101:119–135. https://doi.org/10.1007/s00170-018-2788-x

    Article  Google Scholar 

  11. Popov V, Ahmed S, Shakev N, Topalov A (2019) Gesture-based Interface for real-time control of a Mitsubishi SCARA robot manipulator. IFAC-Papers Online 52:180–185. https://doi.org/10.1016/j.ifacol.2019.12.469

    Article  Google Scholar 

  12. Coupeté E, Moutarde F, Manitsaris S (2015) Gesture recognition using a depth camera for human robot collaboration on assembly line. Procedia Manuf 3:518–525. https://doi.org/10.1016/j.promfg.2015.07.216

    Article  Google Scholar 

  13. Cserteg T, Erdős G, Horváth G (2018) Assisted assembly process by gesture controlled robots. Procedia CIRP 72:51–56. https://doi.org/10.1016/j.procir.2018.03.028

    Article  Google Scholar 

  14. Yu J, Li M, Zhang X, Zhang T, Zhou X (2021) A Multi-sensor gesture interaction system for human-robot cooperation. In: ICNSC 2021-18th IEEE International Conference on Networking, Sensing and Control: Industry 4.0 and AI. https://doi.org/10.1109/ICNSC52481.2021.9702166

    Chapter  Google Scholar 

  15. Köpüklü O, Gunduz A, Kose N, Rigoll G (2019) Real-time hand gesture detection and classification using convolutional neural networks. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition, pp 1–8. https://doi.org/10.1109/FG.2019.8756576

    Chapter  Google Scholar 

  16. Liu H, Fang T, Zhou T, Wang L (2018) Towards robust human-robot collaborative manufacturing: multimodal fusion. IEEE Access 6:P74762–P74771. https://doi.org/10.1109/ACCESS.2018.2884793

    Article  Google Scholar 

  17. Lenz I, Lee H, Saxena A (2015) Deep learning for detecting robotic grasps. Int J Robot Res 34:705–724. https://doi.org/10.48550/arXiv.1301.3592

    Article  Google Scholar 

  18. Duan JG, Fang Y, Zhang QL, Qin JY (2023) HRC for dual-robot intelligent assembly system based on multimodal perception. Proc Inst Mech Eng B J Eng Manuf. https://doi.org/10.1177/09544054231167209

  19. Hashem A, Idris M, Ahmad A (2018) Comparative study of different binarization methods through their effects in characters localization in scene images. Data Knowl Eng 117:216–224. https://doi.org/10.1016/j.datak.2018.07.011

    Article  Google Scholar 

  20. Duan JG, Ma TY, Zhang QL, Qin JY (2021) Design and application of digital twin system for the blade-rotor test rig. J Intell Manuf 34:753–769. https://doi.org/10.1007/s10845-021-01824-w

    Article  Google Scholar 

  21. Mazhar O, Navarro B, Ramdani S, Passama R, Cherubini A (2019) A real-time human-robot interaction framework with robust background invariant hand gesture detection. Robot Comput Integr Manuf 60:34–48. https://doi.org/10.1016/j.rcim.2019.05.008

    Article  Google Scholar 

  22. Magrini E, Ferraguti F, Ronga AJ, Pini F, Luca AD, Leali F (2020) Human-robot coexistence and interaction in open industrial cells. Robot Comput Integr Manuf 61:101846. https://doi.org/10.1016/j.rcim.2019.101846

    Article  Google Scholar 

Download references

Code availability

Not applicable.

Author information

Authors and Affiliations

  1. China Institute of FTZ Supply Chain, Shanghai Maritime University, Shanghai, China

    Jianguo Duan, Qinglei Zhang & Jiyun Qin

  2. Logistics Engineering College, Shanghai Maritime University, Shanghai, China

    Yuan Fang

Authors
  1. Jianguo Duan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuan Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qinglei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jiyun Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Jianguo Duan: conceptualization, supervision, validation. Yuan Fang: investigation, methodology, software, writing—original draft, visualization. Qinglei Zhang: writing—reviewing and editing, resources, project administration. Jiyun Qin: writing—reviewing and editing.

Corresponding author

Correspondence to Jianguo Duan.

Ethics declarations

Ethics approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

This work has been approved by all authors for publication.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Duan, J., Fang, Y., Zhang, Q. et al. HRC of intelligent assembly system based on multimodal gesture control. Int J Adv Manuf Technol 127, 4307–4319 (2023). https://doi.org/10.1007/s00170-023-11804-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-023-11804-4

Keywords

Search

Navigation