Skip to main content

Advertisement

SpringerLink
Log in

Multimodal perception-fusion-control and human–robot collaboration in manufacturing: a review

  • Critical Review
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

Collaborative robots, also known as cobots, are designed to work alongside humans in a shared workspace and provide assistance to them. With the rapid development of robotics and artificial intelligence in recent years, cobots have become faster, smarter, more accurate, and more dependable. They have found applications in a broad range of scenarios where humans require assistance, such as in the home, healthcare, and manufacturing. In manufacturing, in particular, collaborative robots combine the precision and strength of robots with the flexibility of human dexterity to replace or aid humans in highly repetitive or hazardous manufacturing tasks. However, human–robot interaction still needs improvement in terms of adaptability, decision making, and robustness to changing scenarios and uncertainty, especially in the context of continuous interaction with human operators. Collaborative robots and humans must establish an intuitive and understanding rapport to build a cooperative working relationship. Therefore, human–robot interaction is a crucial research problem in robotics. This paper provides a summary of the research on human–robot interaction over the past decade, with a focus on interaction methods in human–robot collaboration, environment perception, task allocation strategies, and scenarios for human–robot collaboration in manufacturing. Finally, the paper presents the primary research directions and challenges for the future development of collaborative robots.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

Data availability

All data generated or analyzed during this study are included in the present article.

References

  1. Magrini E, Ferraguti F, Ronga AJ et al (2020) Human-robot coexistence and interaction in open industrial cells. Robotics and Computer-Integrated Manufacturing 61:101846. https://doi.org/10.1016/j.rcim.2019.101846

    Article  Google Scholar 

  2. Murali PK, Darvish K, Mastrogiovanni F (2020) Deployment and evaluation of a flexible human–robot collaboration model based on AND/OR graphs in a manufacturing environment. Intel Serv Robotics 13:439–457. https://doi.org/10.1007/s11370-020-00332-9

    Article  Google Scholar 

  3. Isma A, Brahim B (2015) Time-dependant trajectory generation for tele-operated mobile manipulator. 2015 3rd International Conference on Control, Engineering & Information Technology (CEIT). IEEE, Tlemcen, Algeria, pp 1–5

    Google Scholar 

  4. Hvilshøj M, Bøgh S, Skov Nielsen O, Madsen O (2012) Autonomous industrial mobile manipulation (AIMM): past, present and future. Industrial Robot: An International Journal 39:120–135. https://doi.org/10.1108/01439911211201582

    Article  Google Scholar 

  5. Stadler S, Weiss A, Mirnig N, Tscheligi M (2013) Anthropomorphism in the factory—a paradigm change? 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, Tokyo, Japan, pp 231–232

    Chapter  Google Scholar 

  6. Peshkin MA, Colgate JE, Wannasuphoprasit W et al (2001) Cobot architecture IEEE Trans Robot Automat 17:377–390. https://doi.org/10.1109/70.954751

    Article  Google Scholar 

  7. Müller R, Vette M, Geenen A (2017) Skill-based dynamic task allocation in human-robot-cooperation with the example of welding application. Procedia Manufacturing 11:13–21. https://doi.org/10.1016/j.promfg.2017.07.113

    Article  Google Scholar 

  8. Schiavi R, Bicchi A, Flacco F (2009) Integration of active and passive compliance control for safe human-robot coexistence. 2009 IEEE International Conference on Robotics and Automation. IEEE, Kobe, pp 259–264

    Chapter  Google Scholar 

  9. Wang L, Gao R, Váncza J et al (2019) Symbiotic human-robot collaborative assembly. CIRP Ann 68:701–726. https://doi.org/10.1016/j.cirp.2019.05.002

    Article  Google Scholar 

  10. Müller R, Vette M, Mailahn O (2016) Process-oriented task assignment for assembly processes with human-robot interaction. Procedia CIRP 44:210–215. https://doi.org/10.1016/j.procir.2016.02.080

    Article  Google Scholar 

  11. Barbazza L, Faccio M, Oscari F, Rosati G (2017) Agility in assembly systems: a comparison model. AA 37:411–421

    Article  Google Scholar 

  12. Gualtieri L, Rauch E, Vidoni R, Matt DT (2020) Safety, ergonomics and efficiency in human-robot collaborative assembly: design guidelines and requirements. Procedia CIRP 91:367–372. https://doi.org/10.1016/j.procir.2020.02.188

    Article  Google Scholar 

  13. Schmidtler J, Knott V, Hölzel C, Bengler K (2015) Human centered assistance applications for the working environment of the future. OER 12:83–95. https://doi.org/10.3233/OER-150226

    Article  Google Scholar 

  14. Gervasi R, Mastrogiacomo L, Franceschini F (2020) A conceptual framework to evaluate human-robot collaboration. Int J Adv Manuf Technol 108:841–865. https://doi.org/10.1007/s00170-020-05363-1

    Article  Google Scholar 

  15. Liu Q, Liu Z, Xu W et al (2019) Human-robot collaboration in disassembly for sustainable manufacturing. Int J Prod Res 57:4027–4044. https://doi.org/10.1080/00207543.2019.1578906

    Article  Google Scholar 

  16. Nikolakis N, Maratos V, Makris S (2019) A cyber physical system (CPS) approach for safe human-robot collaboration in a shared workplace. Robotics and Computer-Integrated Manufacturing 56:233–243. https://doi.org/10.1016/j.rcim.2018.10.003

    Article  Google Scholar 

  17. Prati E, Peruzzini M, Pellicciari M, Raffaeli R (2021) How to include user experience in the design of human-robot interaction. Robotics and Computer-Integrated Manufacturing 68:102072. https://doi.org/10.1016/j.rcim.2020.102072

    Article  Google Scholar 

  18. Jahanmahin R, Masoud S, Rickli J, Djuric A (2022) Human-robot interactions in manufacturing: a survey of human behavior modeling. Robotics and Computer-Integrated Manufacturing 78:102404. https://doi.org/10.1016/j.rcim.2022.102404

    Article  Google Scholar 

  19. Rogowski A (2012) Industrially oriented voice control system. Robotics and Computer-Integrated Manufacturing 28:303–315. https://doi.org/10.1016/j.rcim.2011.09.010

    Article  Google Scholar 

  20. Quek F, McNeill D, Bryll R et al (2002) Multimodal human discourse: gesture and speech. ACM Trans Comput-Hum Interact 9:171–193. https://doi.org/10.1145/568513.568514

    Article  Google Scholar 

  21. Cutugno F, Leano VA, Rinaldi R, Mignini G (2012) Multimodal framework for mobile interaction. In: Proceedings of the International Working Conference on Advanced Visual Interfaces. ACM, Capri Island Italy, pp 197–203

  22. Villani V, Pini F, Leali F, Secchi C (2018) Survey on human–robot collaboration in industrial settings: safety, intuitive interfaces and applications. Mechatronics 55:248–266. https://doi.org/10.1016/j.mechatronics.2018.02.009

    Article  Google Scholar 

  23. Barattini P, Morand C, Robertson NM (2012) A proposed gesture set for the control of industrial collaborative robots. 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication. IEEE, Paris, France, pp 132–137

    Google Scholar 

  24. Lee SJ, Kang BO, Chung H, Lee Y (2014) Intra- and inter-frame features for automatic speech recognition. ETRI J 36:514–517. https://doi.org/10.4218/etrij.14.0213.0181

    Article  Google Scholar 

  25. Lee K, Joe H, Lim H et al (2021) Sequential routing framework: fully capsule network-based speech recognition. Comput Speech Lang 70:101228. https://doi.org/10.1016/j.csl.2021.101228

    Article  Google Scholar 

  26. Lin P-C, Yankson B, Chauhan V, Tsukada M (2022) Building a speech recognition system with privacy identification information based on Google Voice for social robots. J Supercomput 78:15060–15088. https://doi.org/10.1007/s11227-022-04487-3

    Article  Google Scholar 

  27. Lu L, Kanda N, Li J, Gong Y (2021) Streaming end-to-end multi-talker speech recognition. IEEE Signal Process Lett 28:803–807. https://doi.org/10.1109/LSP.2021.3070817

    Article  Google Scholar 

  28. Aldarmaki H, Ullah A, Ram S, Zaki N (2022) Unsupervised automatic speech recognition: a review. Speech Commun 139:76–91. https://doi.org/10.1016/j.specom.2022.02.005

    Article  Google Scholar 

  29. Birch B, Griffiths C, Morgan A (2021) Environmental effects on reliability and accuracy of MFCC based voice recognition for industrial human-robot-interaction. Proceedings of the Institution of Mechanical Engineers, Part B: Journal of Engineering Manufacture 235:1939–1948. https://doi.org/10.1177/09544054211014492

    Article  Google Scholar 

  30. Li D, Sun L, Xu X et al (2021) BLSTM and CNN stacking architecture for speech emotion recognition. Neural Process Lett 53:4097–4115. https://doi.org/10.1007/s11063-021-10581-z

    Article  Google Scholar 

  31. Van Trinh L, Le Dao Thi T, Le Xuan T, Castelli E (2022) Emotional speech recognition using deep neural networks. Sensors 22:1414. https://doi.org/10.3390/s22041414

    Article  Google Scholar 

  32. Brammer A, Laroche C (2012) Noise and communication: a three-year update. Noise Health 14:281. https://doi.org/10.4103/1463-1741.104894

    Article  Google Scholar 

  33. Yang R, Sarkar S, Loeding B (2010) Handling movement epenthesis and hand segmentation ambiguities in continuous sign language recognition using nested dynamic programming. IEEE Trans Pattern Anal Mach Intell 32:462–477. https://doi.org/10.1109/TPAMI.2009.26

    Article  Google Scholar 

  34. Liu H, Wang L (2018) Gesture recognition for human-robot collaboration: a review. Int J Ind Ergon 68:355–367. https://doi.org/10.1016/j.ergon.2017.02.004

    Article  Google Scholar 

  35. Li Y, Miao Q, Tian K et al (2019) Large-scale gesture recognition with a fusion of RGB-D data based on optical flow and the C3D model. Pattern Recogn Lett 119:187–194. https://doi.org/10.1016/j.patrec.2017.12.003

    Article  Google Scholar 

  36. Zhang Y, Shi L, Wu Y et al (2020) Gesture recognition based on deep deformable 3D convolutional neural networks. Pattern Recogn 107:107416. https://doi.org/10.1016/j.patcog.2020.107416

    Article  Google Scholar 

  37. Mazhar O, Navarro B, Ramdani S et al (2019) A real-time human-robot interaction framework with robust background invariant hand gesture detection. Robotics and Computer-Integrated Manufacturing 60:34–48. https://doi.org/10.1016/j.rcim.2019.05.008

    Article  Google Scholar 

  38. Cserteg T, Erdős G, Horváth G (2018) Assisted assembly process by gesture controlled robots. Procedia CIRP 72:51–56. https://doi.org/10.1016/j.procir.2018.03.028

    Article  Google Scholar 

  39. Turk M (2014) Multimodal interaction: a review. Pattern Recogn Lett 36:189–195. https://doi.org/10.1016/j.patrec.2013.07.003

    Article  Google Scholar 

  40. Karpov AA, Yusupov RM (2018) Multimodal interfaces of human–computer interaction. Her Russ Acad Sci 88:67–74. https://doi.org/10.1134/S1019331618010094

    Article  Google Scholar 

  41. Bongers B, Van Der Veer GC (2007) Towards a multimodal interaction space: categorisation and applications. Pers Ubiquit Comput 11:609–619. https://doi.org/10.1007/s00779-006-0138-8

    Article  Google Scholar 

  42. Duan J, Fang Y, Zhang Q, Qin J (2023) HRC for dual-robot intelligent assembly system based on multimodal perception. In: Proceedings of the Institution of Mechanical Engineers, Part B: J Eng Manuf. https://doi.org/10.1177/09544054231167209

  43. Debie E, Fernandez Rojas R, Fidock J et al (2021) Multimodal fusion for objective assessment of cognitive workload: a review. IEEE Trans Cybern 51:1542–1555. https://doi.org/10.1109/TCYB.2019.2939399

    Article  Google Scholar 

  44. Suk H-I, Lee S-W, Shen D (2014) Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis. Neuroimage 101:569–582. https://doi.org/10.1016/j.neuroimage.2014.06.077

    Article  Google Scholar 

  45. Liu H, Fang T, Zhou T, Wang L (2018) Towards robust human-robot collaborative manufacturing: multimodal fusion. IEEE Access 6:74762–74771. https://doi.org/10.1109/ACCESS.2018.2884793

    Article  Google Scholar 

  46. Rouast PV, Adam MTP, Chiong R (2021) Deep learning for human affect recognition: insights and new developments. IEEE Trans Affective Comput 12:524–543. https://doi.org/10.1109/TAFFC.2018.2890471

    Article  Google Scholar 

  47. Mamyrbayev OZh, Alimhan K, Amirgaliyev B et al (2020) Multimodal systems for speech recognition IJMC 18:314. https://doi.org/10.1504/IJMC.2020.107097

    Article  Google Scholar 

  48. Lian Z, Liu B, Tao J (2021) CTNet: Conversational transformer network for emotion recognition. IEEE/ACM Trans Audio Speech Lang Process 29:985–1000. https://doi.org/10.1109/TASLP.2021.3049898

    Article  Google Scholar 

  49. Mohammed A, Wang L (2018) Brainwaves driven human-robot collaborative assembly. CIRP Ann 67:13–16. https://doi.org/10.1016/j.cirp.2018.04.048

    Article  Google Scholar 

  50. Wang L, Liu S, Cooper C et al (2021) Function block-based human-robot collaborative assembly driven by brainwaves. CIRP Ann 70:5–8. https://doi.org/10.1016/j.cirp.2021.04.091

    Article  Google Scholar 

  51. Riseborough MG (1981) Physiographic gestures as decoding facilitators: three experiments exploring a neglected facet of communication. J Nonverbal Behav 5:172–183. https://doi.org/10.1007/BF00986134

    Article  Google Scholar 

  52. Long P, Chevallereau C, Chablat D, Girin A (2018) An industrial security system for human-robot coexistence. IR 45:220–226. https://doi.org/10.1108/IR-09-2017-0165

    Article  Google Scholar 

  53. Vasic M, Billard A (2013) Safety issues in human-robot interactions. 2013 IEEE International Conference on Robotics and Automation. IEEE, Karlsruhe, Germany, pp 197–204

    Chapter  Google Scholar 

  54. Rosenstrauch MJ, Kruger J (2017) Safe human-robot-collaboration-introduction and experiment using ISO/TS 15066. 2017 3rd International Conference on Control, Automation and Robotics (ICCAR). IEEE, Nagoya, Japan, pp 740–744

    Chapter  Google Scholar 

  55. Aivaliotis P, Aivaliotis S, Gkournelos C et al (2019) Power and force limiting on industrial robots for human-robot collaboration. Robotics and Computer-Integrated Manufacturing 59:346–360. https://doi.org/10.1016/j.rcim.2019.05.001

    Article  Google Scholar 

  56. Yao B, Zhou Z, Wang L et al (2018) Sensor-less external force detection for industrial manipulators to facilitate physical human-robot interaction. J Mech Sci Technol 32:4909–4923. https://doi.org/10.1007/s12206-018-0939-5

    Article  Google Scholar 

  57. Yao B, Zhou Z, Wang L et al (2018) Sensorless and adaptive admittance control of industrial robot in physical human−robot interaction. Robotics and Computer-Integrated Manufacturing 51:158–168. https://doi.org/10.1016/j.rcim.2017.12.004

    Article  Google Scholar 

  58. Maurtua I, Ibarguren A, Kildal J et al (2017) Human–robot collaboration in industrial applications: safety, interaction and trust. Int J Adv Rob Syst 14:172988141771601. https://doi.org/10.1177/1729881417716010

    Article  Google Scholar 

  59. Pang G, Deng J, Wang F et al (2018) Development of flexible robot skin for safe and natural human–robot collaboration. Micromachines 9:576. https://doi.org/10.3390/mi9110576

    Article  Google Scholar 

  60. Ding Y, Thomas U (2020) Collision avoidance with proximity servoing for redundant serial robot manipulators. 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, Paris, France, pp 10249–10255

    Chapter  Google Scholar 

  61. Al-Yacoub A, Zhao YC, Eaton W et al (2021) Improving human robot collaboration through force/torque based learning for object manipulation. Robotics and Computer-Integrated Manufacturing 69:102111. https://doi.org/10.1016/j.rcim.2020.102111

    Article  Google Scholar 

  62. Fabrizio F, De Luca A (2017) Real-time computation of distance to dynamic obstacles with multiple depth sensors. IEEE Robot Autom Lett 2:56–63. https://doi.org/10.1109/LRA.2016.2535859

    Article  Google Scholar 

  63. Zabalza J, Fei Z, Wong C et al (2019) Smart sensing and adaptive reasoning for enabling industrial robots with interactive human-robot capabilities in dynamic environments—a case study. Sensors 19:1354. https://doi.org/10.3390/s19061354

    Article  Google Scholar 

  64. Wang X, Yang C, Ju Z et al (2017) Robot manipulator self-identification for surrounding obstacle detection. Multimed Tools Appl 76:6495–6520. https://doi.org/10.1007/s11042-016-3275-8

    Article  Google Scholar 

  65. Kolar P, Benavidez P, Jamshidi M (2020) Survey of datafusion techniques for laser and vision based sensor integration for autonomous navigation. Sensors 20:2180. https://doi.org/10.3390/s20082180

    Article  Google Scholar 

  66. Su H, Sandoval J, Vieyres P et al (2018) Safety-enhanced collaborative framework for tele-operated minimally invasive surgery using a 7-DoF torque-controlled robot. Int J Control Autom Syst 16:2915–2923. https://doi.org/10.1007/s12555-017-0486-3

    Article  Google Scholar 

  67. Liu H, Wang L (2017) Human motion prediction for human-robot collaboration. J Manuf Syst 44:287–294. https://doi.org/10.1016/j.jmsy.2017.04.009

    Article  Google Scholar 

  68. Zanchettin AM, Casalino A, Piroddi L, Rocco P (2019) Prediction of human activity patterns for human–robot collaborative assembly tasks. IEEE Trans Ind Inf 15:3934–3942. https://doi.org/10.1109/TII.2018.2882741

    Article  Google Scholar 

  69. Wang W, Li R, Chen Y et al (2022) Predicting human intentions in human–robot hand-over tasks through multimodal learning. IEEE Trans Automat Sci Eng 19:2339–2353. https://doi.org/10.1109/TASE.2021.3074873

    Article  Google Scholar 

  70. Zhang J, Liu H, Chang Q et al (2020) Recurrent neural network for motion trajectory prediction in human-robot collaborative assembly. CIRP Ann 69:9–12. https://doi.org/10.1016/j.cirp.2020.04.077

    Article  Google Scholar 

  71. Kenk M, Hassaballah M, Brethé J-F (2019) Human-aware Robot Navigation in Logistics Warehouses: In: Proceedings of the 16th International Conference on Informatics in Control, Automation and Robotics. SCITEPRESS - Science and Technology Publications, Prague, Czech Republic, pp 371–378

  72. Zhang J, Wang P, Gao RX (2021) Hybrid machine learning for human action recognition and prediction in assembly. Robotics and Computer-Integrated Manufacturing 72:102184. https://doi.org/10.1016/j.rcim.2021.102184

    Article  Google Scholar 

  73. Lasota PA, Rossano GF, Shah JA (2014) Toward safe close-proximity human-robot interaction with standard industrial robots. 2014 IEEE International Conference on Automation Science and Engineering (CASE). IEEE, Taipei, pp 339–344

    Chapter  Google Scholar 

  74. Zanchettin AM, Ceriani NM, Rocco P et al (2016) Safety in human-robot collaborative manufacturing environments: metrics and control. IEEE Trans Automat Sci Eng 13:882–893. https://doi.org/10.1109/TASE.2015.2412256

    Article  Google Scholar 

  75. Matsas E, Vosniakos G-C, Batras D (2018) Prototyping proactive and adaptive techniques for human-robot collaboration in manufacturing using virtual reality. Robotics and Computer-Integrated Manufacturing 50:168–180. https://doi.org/10.1016/j.rcim.2017.09.005

    Article  Google Scholar 

  76. Zhao X, He Y, Chen X, Liu Z (2021) Human–robot collaborative assembly based on eye-hand and a finite state machine in a virtual environment. Appl Sci 11:5754. https://doi.org/10.3390/app11125754

    Article  Google Scholar 

  77. Khatib M, Al Khudir K, De Luca A (2021) Human-robot contactless collaboration with mixed reality interface. Robotics and Computer-Integrated Manufacturing 67:102030. https://doi.org/10.1016/j.rcim.2020.102030

    Article  Google Scholar 

  78. Xiao J, Wang P, Lu H, Zhang H (2020) A three-dimensional mapping and virtual reality-based human–robot interaction for collaborative space exploration. Int J Adv Rob Syst 17:172988142092529. https://doi.org/10.1177/1729881420925293

    Article  Google Scholar 

  79. Wang Q, Cheng Y, Jiao W et al (2019) Virtual reality human-robot collaborative welding: a case study of weaving gas tungsten arc welding. J Manuf Process 48:210–217. https://doi.org/10.1016/j.jmapro.2019.10.016

    Article  Google Scholar 

  80. Malik AA, Masood T, Bilberg A (2020) Virtual reality in manufacturing: immersive and collaborative artificial-reality in design of human-robot workspace. Int J Comput Integr Manuf 33:22–37. https://doi.org/10.1080/0951192X.2019.1690685

    Article  Google Scholar 

  81. Oyekan JO, Hutabarat W, Tiwari A et al (2019) The effectiveness of virtual environments in developing collaborative strategies between industrial robots and humans. Robotics and Computer-Integrated Manufacturing 55:41–54. https://doi.org/10.1016/j.rcim.2018.07.006

    Article  Google Scholar 

  82. Kana S, Lakshminarayanan S, Mohan DM, Campolo D (2021) Impedance controlled human–robot collaborative tooling for edge chamfering and polishing applications. Robotics and Computer-Integrated Manufacturing 72:102199. https://doi.org/10.1016/j.rcim.2021.102199

    Article  Google Scholar 

  83. Lee H, Liau YY, Kim S, Ryu K (2020) Model-based human robot collaboration system for small batch assembly with a virtual fence. Int J of Precis Eng and Manuf-Green Tech 7:609–623. https://doi.org/10.1007/s40684-020-00214-6

    Article  Google Scholar 

  84. Brito T, Lima J, Costa P, Piardi L (2018) Dynamic collision avoidance system for a manipulator based on RGB-D data. In: Ollero A, Sanfeliu A, Montano L et al (eds) ROBOT 2017: Third Iberian Robotics Conference. Springer International Publishing, Cham, pp 643–654

    Chapter  Google Scholar 

  85. Matsas E, Vosniakos G-C (2017) Design of a virtual reality training system for human–robot collaboration in manufacturing tasks. Int J Interact Des Manuf 11:139–153. https://doi.org/10.1007/s12008-015-0259-2

    Article  Google Scholar 

  86. Su H, Ovur SE, Li Z et al (2020) Internet of Things (IoT)-based collaborative control of a redundant manipulator for teleoperated minimally invasive surgeries. 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, Paris, France, pp 9737–9742

    Chapter  Google Scholar 

  87. Kousi N, Gkournelos C, Aivaliotis S et al (2021) Digital twin for designing and reconfiguring human–robot collaborative assembly lines. Appl Sci 11:4620. https://doi.org/10.3390/app11104620

    Article  Google Scholar 

  88. Lv Q, Zhang R, Sun X et al (2021) A digital twin-driven human-robot collaborative assembly approach in the wake of COVID-19. J Manuf Syst 60:837–851. https://doi.org/10.1016/j.jmsy.2021.02.011

    Article  Google Scholar 

  89. Li X, He B, Zhou Y, Li G (2021) Multisource model-driven digital twin system of robotic assembly. IEEE Syst J 15:114–123. https://doi.org/10.1109/JSYST.2019.2958874

    Article  Google Scholar 

  90. Bilberg A, Malik AA (2019) Digital twin driven human–robot collaborative assembly. CIRP Ann 68:499–502. https://doi.org/10.1016/j.cirp.2019.04.011

    Article  Google Scholar 

  91. Duan J-G, Ma T-Y, Zhang Q-L et al (2023) Design and application of digital twin system for the blade-rotor test rig. J Intell Manuf 34:753–769. https://doi.org/10.1007/s10845-021-01824-w

    Article  Google Scholar 

  92. Liu Z, Zhang Q, Duan J, Liu D (2022) Digital twin–based testing process management for large and complex equipment components. Int J Adv Manuf Technol 121:3143–3161. https://doi.org/10.1007/s00170-022-09475-8

    Article  Google Scholar 

  93. Duan J, Gong X, Zhang Q, Qin J (2023) A digital twin–driven monitoring framework for dual-robot collaborative manipulation. Int J Adv Manuf Technol 125:4579–4599. https://doi.org/10.1007/s00170-023-11064-2

    Article  Google Scholar 

  94. Wang XV, Kemény Z, Váncza J, Wang L (2017) Human–robot collaborative assembly in cyber-physical production: classification framework and implementation. CIRP Ann 66:5–8. https://doi.org/10.1016/j.cirp.2017.04.101

    Article  Google Scholar 

  95. Lamon E, De Franco A, Peternel L, Ajoudani A (2019) A capability-aware role allocation approach to industrial assembly tasks. IEEE Robot Autom Lett 4:3378–3385. https://doi.org/10.1109/LRA.2019.2926963

    Article  Google Scholar 

  96. Michalos G, Kousi N, Karagiannis P et al (2018) Seamless human robot collaborative assembly—an automotive case study. Mechatronics 55:194–211. https://doi.org/10.1016/j.mechatronics.2018.08.006

    Article  Google Scholar 

  97. Tsarouchi P, Matthaiakis A-S, Makris S, Chryssolouris G (2017) On a human-robot collaboration in an assembly cell. Int J Comput Integr Manuf 30:580–589. https://doi.org/10.1080/0951192X.2016.1187297

    Article  Google Scholar 

  98. Heydaryan S, Suaza Bedolla J, Belingardi G (2018) Safety design and development of a human-robot collaboration assembly process in the automotive industry. Appl Sci 8:344. https://doi.org/10.3390/app8030344

    Article  Google Scholar 

  99. Bruno G, Antonelli D (2018) Dynamic task classification and assignment for the management of human-robot collaborative teams in workcells. Int J Adv Manuf Technol 98:2415–2427. https://doi.org/10.1007/s00170-018-2400-4

    Article  Google Scholar 

  100. Zhang R, Lv Q, Li J et al (2022) A reinforcement learning method for human-robot collaboration in assembly tasks. Robotics and Computer-Integrated Manufacturing 73:102227. https://doi.org/10.1016/j.rcim.2021.102227

    Article  Google Scholar 

  101. Casalino A, Zanchettin AM, Piroddi L, Rocco P (2021) Optimal scheduling of human–robot collaborative assembly operations with time petri nets. IEEE Trans Automat Sci Eng 18:70–84. https://doi.org/10.1109/TASE.2019.2932150

    Article  Google Scholar 

  102. Dalle Mura M, Dini G (2019) Designing assembly lines with humans and collaborative robots: a genetic approach. CIRP Ann 68:1–4. https://doi.org/10.1016/j.cirp.2019.04.006

    Article  Google Scholar 

  103. Yu T, Huang J, Chang Q (2020) Mastering the working sequence in human-robot collaborative assembly based on reinforcement learning. IEEE Access 8:163868–163877. https://doi.org/10.1109/ACCESS.2020.3021904

    Article  Google Scholar 

  104. Gualtieri L, Rauch E, Vidoni R (2021) Methodology for the definition of the optimal assembly cycle and calculation of the optimized assembly cycle time in human-robot collaborative assembly. Int J Adv Manuf Technol 113:2369–2384. https://doi.org/10.1007/s00170-021-06653-y

    Article  Google Scholar 

  105. Liu Z, Liu Q, Wang L et al (2021) Task-level decision-making for dynamic and stochastic human-robot collaboration based on dual agents deep reinforcement learning. Int J Adv Manuf Technol 115:3533–3552. https://doi.org/10.1007/s00170-021-07265-2

    Article  Google Scholar 

  106. Aggarwal K, Singh SK, Chopra M et al (2022) Deep learning in robotics for strengthening Industry 4.0.: opportunities, challenges and future directions. In: Nedjah N, Abd El-Latif AA, Gupta BB, Mourelle LM (eds) Robotics and AI for Cybersecurity and Critical Infrastructure in Smart Cities. Springer International Publishing, Cham, pp 1–19

    Google Scholar 

  107. Guzsvinecz T, Szucs V, Sik-Lanyi C (2019) Suitability of the kinect sensor and leap motion controller—a literature review. Sensors 19:1072. https://doi.org/10.3390/s19051072

    Article  Google Scholar 

  108. Litim E, Elawady I, BELMOKHTAR S, et al (2021) Trajectory planning and motion control schemes for 2DoF planar parallel manipulator biglide type with elastic joints: a comparative study. 08:88. https://doi.org/10.15199/48.2021

  109. Malang Politeknik Negeri, Huda MAN, Susilo SH et al (2022) Implementation of inverse kinematic and trajectory planning on 6-DOF robotic arm for straight-flat welding movement. LOGIC 22:51–61. https://doi.org/10.31940/logic.v22i1.51-61

    Article  Google Scholar 

  110. Palmieri G, Scoccia C (2021) Motion planning and control of redundant manipulators for dynamical obstacle avoidance. Machines 9:121. https://doi.org/10.3390/machines9060121

    Article  Google Scholar 

  111. Gutierrez-Giles A, Evangelista-Hernandez LU, Arteaga MA et al (2021) A force/motion control approach based on trajectory planning for industrial robots with closed control architecture. IEEE Access 9:80728–80740. https://doi.org/10.1109/ACCESS.2021.3085528

    Article  Google Scholar 

  112. Shi L, Yao H, Shan M et al (2022) Robust control of a space robot based on an optimized adaptive variable structure control method. Aerosp Sci Technol 120:107267. https://doi.org/10.1016/j.ast.2021.107267

    Article  Google Scholar 

  113. Liu Y, Zhang X, Qu T et al (2023) Intelligent robot motion trajectory planning based on machine vision. Int J Syst Assur Eng Manag 14:776–785. https://doi.org/10.1007/s13198-021-01559-0

    Article  Google Scholar 

  114. Feng N, Wu S (2023) Research on motion control and trajectory planning algorithm of mobile manipulator based on deep learning. 2023 International Conference on Mechatronics, IoT and Industrial Informatics (ICMIII). IEEE, Melbourne, Australia, pp 271–274

    Chapter  Google Scholar 

  115. Madridano Á, Al-Kaff A, Martín D, De La Escalera A (2021) Trajectory planning for multi-robot systems: methods and applications. Expert Syst Appl 173:114660. https://doi.org/10.1016/j.eswa.2021.114660

    Article  Google Scholar 

  116. Pokle A, Martin-Martin R, Goebel P et al (2019) Deep local trajectory replanning and control for robot navigation. 2019 International Conference on Robotics and Automation (ICRA). IEEE, Montreal, QC, Canada, pp 5815–5822

    Chapter  Google Scholar 

  117. Zhang S, Xia Q, Chen M, Cheng S (2023) Multi-objective optimal trajectory planning for robotic arms using deep reinforcement learning. Sensors 23:5974. https://doi.org/10.3390/s23135974

    Article  Google Scholar 

  118. Qie X, Kang C, Zong G, Chen S (2022) Trajectory planning and simulation study of redundant robotic arm for upper limb rehabilitation based on back propagation neural network and genetic algorithm. Sensors 22:4071. https://doi.org/10.3390/s22114071

    Article  Google Scholar 

  119. Ekrem Ö, Aksoy B (2023) Trajectory planning for a 6-axis robotic arm with particle swarm optimization algorithm. Eng Appl Artif Intell 122:106099. https://doi.org/10.1016/j.engappai.2023.106099

    Article  Google Scholar 

  120. Chai R, Liu D, Liu T et al (2023) Deep learning-based trajectory planning and control for autonomous ground vehicle parking maneuver. IEEE Trans Automat Sci Eng 20:1633–1647. https://doi.org/10.1109/TASE.2022.3183610

    Article  Google Scholar 

  121. Gao Q, Yuan Q, Sun Y, Xu L (2023) Path planning algorithm of robot arm based on improved RRT* and BP neural network algorithm. Journal of King Saud University - Computer and Information Sciences 35:101650. https://doi.org/10.1016/j.jksuci.2023.101650

    Article  Google Scholar 

  122. Meng BH, Godage IS, Kanj I (2022) RRT*-based path planning for continuum arms. IEEE Robot Autom Lett 7:6830–6837. https://doi.org/10.1109/LRA.2022.3174257

    Article  Google Scholar 

  123. Yi J, Yuan Q, Sun R, Bai H (2022) Path planning of a manipulator based on an improved P_RRT* algorithm. Complex Intell Syst 8:2227–2245. https://doi.org/10.1007/s40747-021-00628-y

    Article  Google Scholar 

  124. Zou Q, Du X, Liu Y et al (2023) Dynamic path planning and motion control of microrobotic swarms for mobile target tracking. IEEE Trans Automat Sci Eng 20:2454–2468. https://doi.org/10.1109/TASE.2022.3207289

    Article  Google Scholar 

  125. Feraco S, Luciani S, Bonfitto A et al (2020) A local trajectory planning and control method for autonomous vehicles based on the RRT algorithm. 2020 AEIT International Conference of Electrical and Electronic Technologies for Automotive (AEIT AUTOMOTIVE). IEEE, Turin, Italy, pp 1–6

    Google Scholar 

  126. Qi J, Yuan Q, Wang C et al (2023) Path planning and collision avoidance based on the RRT*FN framework for a robotic manipulator in various scenarios. Complex Intell Syst. https://doi.org/10.1007/s40747-023-01131-2

    Article  Google Scholar 

  127. Jia Chenhui, Liu Yanbin, Xia Xintao (2009) Research and application of digital assembly process planning and simulative validation. 2009 International Conference on Mechatronics and Automation. IEEE, Changchun, China, pp 2203–2207

    Google Scholar 

  128. Edmondson NF, Redford AH (2002) Generic flexible assembly system design. Assem Autom 22:139–152. https://doi.org/10.1108/01445150210423189

    Article  Google Scholar 

  129. Sawodny O, Aschemann H, Lahres S (2002) An automated gantry crane as a large workspace robot. Control Eng Pract 10:1323–1338. https://doi.org/10.1016/S0967-0661(02)00097-7

    Article  Google Scholar 

  130. Krüger J, Lien TK, Verl A (2009) Cooperation of human and machines in assembly lines. CIRP Ann 58:628–646. https://doi.org/10.1016/j.cirp.2009.09.009

    Article  Google Scholar 

  131. Hashemi-Petroodi SE, Thevenin S, Kovalev S, Dolgui A (2020) Operations management issues in design and control of hybrid human-robot collaborative manufacturing systems: a survey. Annu Rev Control 49:264–276. https://doi.org/10.1016/j.arcontrol.2020.04.009

    Article  Google Scholar 

  132. Mateus JC, Claeys D, Limère V et al (2019) A structured methodology for the design of a human-robot collaborative assembly workplace. Int J Adv Manuf Technol 102:2663–2681. https://doi.org/10.1007/s00170-019-03356-3

    Article  Google Scholar 

  133. Pellegrinelli S, Moro FL, Pedrocchi N et al (2016) A probabilistic approach to workspace sharing for human–robot cooperation in assembly tasks. CIRP Ann 65:57–60. https://doi.org/10.1016/j.cirp.2016.04.035

    Article  Google Scholar 

  134. Sadrfaridpour B, Wang Y (2018) Collaborative assembly in hybrid manufacturing cells: an integrated framework for human–robot interaction. IEEE Trans Automat Sci Eng 15:1178–1192. https://doi.org/10.1109/TASE.2017.2748386

    Article  Google Scholar 

  135. Cherubini A, Passama R, Crosnier A et al (2016) Collaborative manufacturing with physical human–robot interaction. Robotics and Computer-Integrated Manufacturing 40:1–13. https://doi.org/10.1016/j.rcim.2015.12.007

    Article  Google Scholar 

  136. Morioka M, Sakakibara S (2010) A new cell production assembly system with human–robot cooperation. CIRP Ann 59:9–12. https://doi.org/10.1016/j.cirp.2010.03.044

    Article  Google Scholar 

  137. Sun Y, Wang W, Chen Y, Jia Y (2022) Learn how to assist humans through human teaching and robot learning in human–robot collaborative assembly. IEEE Trans Syst Man Cybern, Syst 52:728–738. https://doi.org/10.1109/TSMC.2020.3005340

    Article  Google Scholar 

  138. Zhou Z, Liu J, Pham DT et al (2019) Disassembly sequence planning: recent developments and future trends. Proceedings of the Institution of Mechanical Engineers, Part B: Journal of Engineering Manufacture 233:1450–1471. https://doi.org/10.1177/0954405418789975

    Article  Google Scholar 

  139. Xu W, Tang Q, Liu J et al (2020) Disassembly sequence planning using discrete Bees algorithm for human-robot collaboration in remanufacturing. Robotics and Computer-Integrated Manufacturing 62:101860. https://doi.org/10.1016/j.rcim.2019.101860

    Article  Google Scholar 

  140. Liu J, Zhou Z, Pham DT et al (2018) Robotic disassembly sequence planning using enhanced discrete bees algorithm in remanufacturing. Int J Prod Res 56:3134–3151. https://doi.org/10.1080/00207543.2017.1412527

    Article  Google Scholar 

  141. Chen WH, Foo G, Kara S, Pagnucco M (2021) Automated generation and execution of disassembly actions. Robotics and Computer-Integrated Manufacturing 68:102056. https://doi.org/10.1016/j.rcim.2020.102056

    Article  Google Scholar 

  142. Liu Z, Liu Q, Xu W et al (2019) Deep Learning-based human motion prediction considering context awareness for human-robot collaboration in manufacturing. Procedia CIRP 83:272–278. https://doi.org/10.1016/j.procir.2019.04.080

    Article  Google Scholar 

  143. Bdiwi M, Rashid A, Putz M (2016) Autonomous disassembly of electric vehicle motors based on robot cognition. 2016 IEEE International Conference on Robotics and Automation (ICRA). IEEE, Stockholm, Sweden, pp 2500–2505

    Chapter  Google Scholar 

  144. Petković T, Puljiz D, Marković I, Hein B (2019) Human intention estimation based on hidden Markov model motion validation for safe flexible robotized warehouses. Robotics and Computer-Integrated Manufacturing 57:182–196. https://doi.org/10.1016/j.rcim.2018.11.004

    Article  Google Scholar 

  145. Popović G, Cvišić I, Écorchard G et al (2022) Human localization in robotized warehouses based on stereo odometry and ground-marker fusion. Robotics and Computer-Integrated Manufacturing 73:102241. https://doi.org/10.1016/j.rcim.2021.102241

    Article  Google Scholar 

  146. Srinivas S, Yu S (2022) Collaborative order picking with multiple pickers and robots: Integrated approach for order batching, sequencing and picker-robot routing. Int J Prod Econ 254:108634. https://doi.org/10.1016/j.ijpe.2022.108634

    Article  Google Scholar 

  147. Winkelhaus S, Zhang M, Grosse EH, Glock CH (2022) Hybrid order picking: a simulation model of a joint manual and autonomous order picking system. Comput Ind Eng 167:107981. https://doi.org/10.1016/j.cie.2022.107981

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Logistics Engineering College, Shanghai Maritime University, Shanghai, 201306, China

    Jianguo Duan, Liwen Zhuang, Qinglei Zhang, Ying Zhou & Jiyun Qin

Authors
  1. Jianguo Duan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liwen Zhuang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qinglei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ying Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jiyun Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jianguo Duan.

Ethics declarations

Ethics approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

This work has been approved by all authors for publication.

Competing interests

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Duan, J., Zhuang, L., Zhang, Q. et al. Multimodal perception-fusion-control and human–robot collaboration in manufacturing: a review. Int J Adv Manuf Technol 132, 1071–1093 (2024). https://doi.org/10.1007/s00170-024-13385-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-024-13385-2

Keywords

Search

Navigation