Abstract
Human-robot collaboration as a multidisciplinary research topic is still pursuing the robots’ enhanced intelligence to be more human-compatible and fit the dynamic and stochastic characteristics of human. However, the uncertainties brought by the human partner challenge the task-planning and decision-making of the robot. When aiming at industrial tasks like collaborative assembly, dynamics on temporal dimension and stochasticities on the order of procedures need to be further considered. In this work, we bring a new perspective and solution based on reinforcement learning, where the problem is regarded as training an agent towards tasks in dynamic and stochastic environments. Concretely, an adapted training approach based on the deep Q learning method is proposed. This method regards both the robot and the human as the agents in the interactive training environment for deep reinforcement learning. With the consideration of task-level industrial human-robot collaboration, the training logic and the agent-environment interaction have been proposed. For the human-robot collaborative assembly tasks in the case study, it is illustrated that our method could drive the robot represented by one agent to collaborate with the human partner even the human performs randomly on the task procedures.
Similar content being viewed by others
Data availability
Data was uploaded as supplementary materials.
References
Gervasi R, Mastrogiaconno L, Franceschini F (2020) A conceptual framework to evaluate human-robot collaboration. Int J Adv Manuf Technol 108(3):841–865. https://doi.org/10.1007/s00170-020-05363-1
Perez L, Rodriguez-Jimenez S, Rodriguez N, Usamentiaga R, Garcia DF, Wang L (2020) Symbiotic human-robot collaborative approach for increased productivity and enhanced safety in the aerospace manufacturing industry. Int J Adv Manuf Technol 106(3-4):851–863. https://doi.org/10.1007/s00170-019-04638-6
Wang L, Gao R, Váncza J, Krüger J, Wang XV, Makris S, Chryssolouris G (2019) Symbiotic human-robot collaborative assembly. CIRP Ann 68(2):701–726. https://doi.org/10.1016/j.cirp.2019.05.002
Tsarouchi P, Matthaiakis A-S, Makris S, Chryssolouris G (2017) On a human-robot collaboration in an assembly cell. Int J Comput Integr Manuf 30(6):580–589. https://doi.org/10.1080/0951192X.2016.1187297
Faccio M, Minto R, Rosati G, Bottin M (2020) The influence of the product characteristics on human-robot collaboration: a model for the performance of collaborative robotic assembly. Int J Adv Manuf Technol 106(5-6):2317–2331. https://doi.org/10.1007/s00170-019-04670-6
Hayes B, Scassellati B (2013) Challenges in shared-environment human-robot collaboration. learning 8 (9)
Krueger V, Rovida F, Grossmann B, Petrick R, Crosby M, Charzoule A, Garcia GM, Behnke S, Toscano C, Veiga G (2019) Testing the vertical and cyber-physical integration of cognitive robots in manufacturing. Robot Comput Integr Manuf 57:213–229. https://doi.org/10.1016/j.rcim.2018.11.011
Liu Q, Liu Z, Xu W, Tang Q, Zhou Z, Pham DT (2019) Human-robot collaboration in disassembly for sustainable manufacturing. Int J Prod Res 57(12):4027–4044. https://doi.org/10.1080/00207543.2019.1578906
Michalos G, Makris S, Spiliotopoulos J, Misios I, Tsarouchi P, Chryssolouris G (2014) ROBO-PARTNER: Seamless human-robot cooperation for intelligent, flexible and safe operations in the assembly factories of the future. Procedia CIRP 23:71–76. https://doi.org/10.1016/j.procir.2014.10.079
Mura MD, Dini G (2019) Designing assembly lines with humans and collaborative robots: A genetic approach. CIRP Ann 68(1):1–4. https://doi.org/10.1016/j.cirp.2019.04.006
Karami A-B, Jeanpierre L, Mouaddib A-I (2010) Human-robot collaboration for a shared mission. In: 5th ACM/IEEE International. Conference on Human-Robot Interaction (HRI):155–156
Zhou L, Tao H, Paszke W, Stojanovic V, Yang H (2020) PD-type iterative learning control for uncertain spatially interconnected systems. Mathematics 8(9):1528. https://doi.org/10.3390/math8091528
Tao H, Li X, Paszke W, Stojanovic V, Yang H (2021) Robust PD-type iterative learning control for discrete systems with multiple time-delays subjected to polytopic uncertainty and restricted frequency-domain. Multidim Syst Sign Process 32(2):671–692. https://doi.org/10.1007/s11045-020-00754-9
Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller M (2013) Playing atari with deep reinforcement learning. arXiv preprint arXiv:13125602
Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533. https://doi.org/10.1038/nature14236
Silver D, Schrittwieser J, Simonyan K, Antonoglou I, Huang A, Guez A, Hubert T, Baker L, Lai M, Bolton A (2017) Mastering the game of go without human knowledge. Nature 550(7676):354–359. https://doi.org/10.1038/nature24270
Wang L (2019) From intelligence science to intelligent manufacturing. Engineering 5(4):615–618. https://doi.org/10.1016/j.eng.2019.04.011
Liu Z, Wang X, Cai Y, Xu W, Liu Q, Zhou Z, Pham DT (2020) Dynamic risk assessment and active response strategy for industrial human-robot collaboration. Comput Ind Eng 141:106302. https://doi.org/10.1016/j.cie.2020.106302
Sheridan TB (2016) Human–robot interaction: status and challenges. Hum Factors 58(4):525–532. https://doi.org/10.1177/0018720816644364
Chakraborti T, Kambhampati S, Scheutz M, Zhang Y (2017) Ai challenges in human-robot cognitive teaming. arXiv preprint arXiv:170704775
Tsarouchi P, Makris S, Chryssolouris G (2016) Human–robot interaction review and challenges on task planning and programming. Int J Comput Integr Manuf 29(8):916–931. https://doi.org/10.1080/0951192X.2015.1130251
Lasota PA, Shah JA (2015) Analyzing the effects of human-aware motion planning on close-proximity human–robot collaboration. Hum Factors 57(1):21–33. https://doi.org/10.1177/0018720814565188
Pellegrinelli S, Orlandini A, Pedrocchi N, Umbrico A, Tolio T (2017) Motion planning and scheduling for human and industrial-robot collaboration. CIRP Ann 66(1):1–4. https://doi.org/10.1016/j.cirp.2017.04.095
Charalambous G, Fletcher SR, Webb P (2017) The development of a Human Factors Readiness Level tool for implementing industrial human-robot collaboration. Int J Adv Manuf Technol 91(5-8):2465–2475. https://doi.org/10.1007/s00170-016-9876-6
Chen M, Nikolaidis S, Soh H, Hsu D, Srinivasa S (2018) Planning with trust for human-robot collaboration. In: Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction. pp 307-315
Tan JTC, Duan F, Kato R, Arai T, Hall E (2010) Collaboration planning by task analysis in human-robot collaborative manufacturing system. In: Hall E (ed) Advances in Robot Manipulators. IntechOpen, London
Fiore M, Clodic A, Alami R (2016) On planning and task achievement modalities for human-robot collaboration. Experimental Robotics. Springer, In, pp 293–306
Hawkins KP, Vo N, Bansal S, Bobick AF (2013) Probabilistic human action prediction and wait-sensitive planning for responsive human-robot collaboration. In: 13th IEEE-RAS International Conference on Humanoid Robots (Humanoids). pp 499-506
Hayes B, Scassellati B (2016) Autonomously constructing hierarchical task networks for planning and human-robot collaboration. In: IEEE International Conference on Robotics and Automation (ICRA). pp 5469-5476
Johannsmeier L, Haddadin S (2016) A hierarchical human-robot interaction-planning framework for task allocation in collaborative industrial assembly processes. IEEE Rob Autom Lett 2(1):41–48. https://doi.org/10.1109/LRA.2016.2535907
Kardos C, Kovács A, Váncza J (2016) Towards feature-based human-robot assembly process planning. Procedia CIRP 57:516–521. https://doi.org/10.1016/j.procir.2016.11.089
Ranz F, Hummel V, Sihn W (2017) Capability-based task allocation in human-robot collaboration. Procedia Manuf 9:182–189. https://doi.org/10.1016/j.promfg.2017.04.011
Wang XV, Kemény Z, Váncza J, Wang L (2017) Human–robot collaborative assembly in cyber-physical production: Classification framework and implementation. CIRP Ann 66(1):5–8. https://doi.org/10.1016/j.cirp.2017.04.101
Mateus JC, Claeys D, Limere V, Cottyn J, Aghezzaf E (2019) A structured methodology for the design of a human-robot collaborative assembly workplace. Int J Adv Manuf Technol 102(5-8):2663–2681. https://doi.org/10.1007/s00170-019-03356-3
Kumar A, Dimitrakopoulos R, Maulen M (2020) Adaptive self-learning mechanisms for updating short-term production decisions in an industrial mining complex. J Intell Manuf 31(7):1795–1811. https://doi.org/10.1007/s10845-020-01562-5
Xu W, Tang Q, Liu J, Liu Z, Zhou Z, Duc Truong P (2020) Disassembly sequence planning using discrete bees algorithm for human-robot collaboration in remanufacturing. Robot Comput Integr Manuf 62:101860. https://doi.org/10.1016/j.rcim.2019.101860
Wang P, Liu H, Wang L, Gao RX (2018) Deep learning-based human motion recognition for predictive context-aware human-robot collaboration. CIRP Ann 67(1):17–20. https://doi.org/10.1016/j.cirp.2018.04.066
Kartoun U, Stern H, Edan Y (2010) A human-robot collaborative reinforcement learning algorithm. J Intell Robot Syst 60(2):217–239. https://doi.org/10.1007/s10846-010-9422-y
Akkaladevi SC, Plasch M, Maddukuri S, Eitzinger C, Pichler A, Rinner B (2018) Toward an interactive reinforcement based learning framework for human robot collaborative assembly processes. Front Rob AI 5. https://doi.org/10.3389/frobt.2018.00126
Wang W, Li R, Chen Y, Diekel ZM, Jia Y (2019) Facilitating human-robot collaborative tasks by teaching-learning-collaboration from human demonstrations. IEEE Trans Autom Sci Eng 16(2):640–653. https://doi.org/10.1109/tase.2018.2840345
Modares H, Ranatunga I, Lewis FL, Popa DO (2015) Optimized assistive human–robot interaction using reinforcement learning. IEEE Trans Cybern 46(3):655–667. https://doi.org/10.1109/TCYB.2015.2412554
Gu S, Holly E, Lillicrap T, Levine S (2017) Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In: IEEE international conference on robotics and automation (ICRA). pp 3389-3396
Ghadirzadeh A, Butepage J, Maki A, Kragic D, Bjorkman M (2016) A sensorimotor reinforcement learning framework for physical human-robot interaction. 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems.
Chen C, Liu Y, Kreiss S, Alahi A (2019) Crowd-robot interaction: Crowd-aware robot navigation with attention-based deep reinforcement learning. In: 2019 International Conference on Robotics and Automation (ICRA). pp 6015-6022
Trivedi M, Doshi P (2018) Inverse learning of robot behavior for collaborative planning. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp 1-9
Foerster JN (2018) Deep multi-agent reinforcement learning. University of Oxford, Dissertation
Sutton RS, Barto AG (2018) Reinforcement learning: An introduction. MIT press, Cambridge
Code availability
Code is available when required.
Funding
This work is supported by National Natural Science Foundation of China (Grant Nos. 51775399 and 51675389), the Fundamental Research Funds for the Central Universities (WUT: 2020III047), and the KTH-CSC programme of China Scholarship Council and KTH Royal Institute of Technology (No. 201906950003).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Liu, Z., Liu, Q., Wang, L. et al. Task-level decision-making for dynamic and stochastic human-robot collaboration based on dual agents deep reinforcement learning. Int J Adv Manuf Technol 115, 3533–3552 (2021). https://doi.org/10.1007/s00170-021-07265-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00170-021-07265-2