Skip to main content
SpringerLink
Log in

Task-level decision-making for dynamic and stochastic human-robot collaboration based on dual agents deep reinforcement learning

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

Human-robot collaboration as a multidisciplinary research topic is still pursuing the robots’ enhanced intelligence to be more human-compatible and fit the dynamic and stochastic characteristics of human. However, the uncertainties brought by the human partner challenge the task-planning and decision-making of the robot. When aiming at industrial tasks like collaborative assembly, dynamics on temporal dimension and stochasticities on the order of procedures need to be further considered. In this work, we bring a new perspective and solution based on reinforcement learning, where the problem is regarded as training an agent towards tasks in dynamic and stochastic environments. Concretely, an adapted training approach based on the deep Q learning method is proposed. This method regards both the robot and the human as the agents in the interactive training environment for deep reinforcement learning. With the consideration of task-level industrial human-robot collaboration, the training logic and the agent-environment interaction have been proposed. For the human-robot collaborative assembly tasks in the case study, it is illustrated that our method could drive the robot represented by one agent to collaborate with the human partner even the human performs randomly on the task procedures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data availability

Data was uploaded as supplementary materials.

References

  1. Gervasi R, Mastrogiaconno L, Franceschini F (2020) A conceptual framework to evaluate human-robot collaboration. Int J Adv Manuf Technol 108(3):841–865. https://doi.org/10.1007/s00170-020-05363-1

    Article  Google Scholar 

  2. Perez L, Rodriguez-Jimenez S, Rodriguez N, Usamentiaga R, Garcia DF, Wang L (2020) Symbiotic human-robot collaborative approach for increased productivity and enhanced safety in the aerospace manufacturing industry. Int J Adv Manuf Technol 106(3-4):851–863. https://doi.org/10.1007/s00170-019-04638-6

    Article  Google Scholar 

  3. Wang L, Gao R, Váncza J, Krüger J, Wang XV, Makris S, Chryssolouris G (2019) Symbiotic human-robot collaborative assembly. CIRP Ann 68(2):701–726. https://doi.org/10.1016/j.cirp.2019.05.002

    Article  Google Scholar 

  4. Tsarouchi P, Matthaiakis A-S, Makris S, Chryssolouris G (2017) On a human-robot collaboration in an assembly cell. Int J Comput Integr Manuf 30(6):580–589. https://doi.org/10.1080/0951192X.2016.1187297

    Article  Google Scholar 

  5. Faccio M, Minto R, Rosati G, Bottin M (2020) The influence of the product characteristics on human-robot collaboration: a model for the performance of collaborative robotic assembly. Int J Adv Manuf Technol 106(5-6):2317–2331. https://doi.org/10.1007/s00170-019-04670-6

    Article  Google Scholar 

  6. Hayes B, Scassellati B (2013) Challenges in shared-environment human-robot collaboration. learning 8 (9)

  7. Krueger V, Rovida F, Grossmann B, Petrick R, Crosby M, Charzoule A, Garcia GM, Behnke S, Toscano C, Veiga G (2019) Testing the vertical and cyber-physical integration of cognitive robots in manufacturing. Robot Comput Integr Manuf 57:213–229. https://doi.org/10.1016/j.rcim.2018.11.011

    Article  Google Scholar 

  8. Liu Q, Liu Z, Xu W, Tang Q, Zhou Z, Pham DT (2019) Human-robot collaboration in disassembly for sustainable manufacturing. Int J Prod Res 57(12):4027–4044. https://doi.org/10.1080/00207543.2019.1578906

    Article  Google Scholar 

  9. Michalos G, Makris S, Spiliotopoulos J, Misios I, Tsarouchi P, Chryssolouris G (2014) ROBO-PARTNER: Seamless human-robot cooperation for intelligent, flexible and safe operations in the assembly factories of the future. Procedia CIRP 23:71–76. https://doi.org/10.1016/j.procir.2014.10.079

    Article  Google Scholar 

  10. Mura MD, Dini G (2019) Designing assembly lines with humans and collaborative robots: A genetic approach. CIRP Ann 68(1):1–4. https://doi.org/10.1016/j.cirp.2019.04.006

    Article  Google Scholar 

  11. Karami A-B, Jeanpierre L, Mouaddib A-I (2010) Human-robot collaboration for a shared mission. In: 5th ACM/IEEE International. Conference on Human-Robot Interaction (HRI):155–156

  12. Zhou L, Tao H, Paszke W, Stojanovic V, Yang H (2020) PD-type iterative learning control for uncertain spatially interconnected systems. Mathematics 8(9):1528. https://doi.org/10.3390/math8091528

    Article  Google Scholar 

  13. Tao H, Li X, Paszke W, Stojanovic V, Yang H (2021) Robust PD-type iterative learning control for discrete systems with multiple time-delays subjected to polytopic uncertainty and restricted frequency-domain. Multidim Syst Sign Process 32(2):671–692. https://doi.org/10.1007/s11045-020-00754-9

    Article  MathSciNet  MATH  Google Scholar 

  14. Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller M (2013) Playing atari with deep reinforcement learning. arXiv preprint arXiv:13125602

  15. Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533. https://doi.org/10.1038/nature14236

    Article  Google Scholar 

  16. Silver D, Schrittwieser J, Simonyan K, Antonoglou I, Huang A, Guez A, Hubert T, Baker L, Lai M, Bolton A (2017) Mastering the game of go without human knowledge. Nature 550(7676):354–359. https://doi.org/10.1038/nature24270

    Article  Google Scholar 

  17. Wang L (2019) From intelligence science to intelligent manufacturing. Engineering 5(4):615–618. https://doi.org/10.1016/j.eng.2019.04.011

    Article  Google Scholar 

  18. Liu Z, Wang X, Cai Y, Xu W, Liu Q, Zhou Z, Pham DT (2020) Dynamic risk assessment and active response strategy for industrial human-robot collaboration. Comput Ind Eng 141:106302. https://doi.org/10.1016/j.cie.2020.106302

    Article  Google Scholar 

  19. Sheridan TB (2016) Human–robot interaction: status and challenges. Hum Factors 58(4):525–532. https://doi.org/10.1177/0018720816644364

    Article  Google Scholar 

  20. Chakraborti T, Kambhampati S, Scheutz M, Zhang Y (2017) Ai challenges in human-robot cognitive teaming. arXiv preprint arXiv:170704775

  21. Tsarouchi P, Makris S, Chryssolouris G (2016) Human–robot interaction review and challenges on task planning and programming. Int J Comput Integr Manuf 29(8):916–931. https://doi.org/10.1080/0951192X.2015.1130251

    Article  Google Scholar 

  22. Lasota PA, Shah JA (2015) Analyzing the effects of human-aware motion planning on close-proximity human–robot collaboration. Hum Factors 57(1):21–33. https://doi.org/10.1177/0018720814565188

    Article  Google Scholar 

  23. Pellegrinelli S, Orlandini A, Pedrocchi N, Umbrico A, Tolio T (2017) Motion planning and scheduling for human and industrial-robot collaboration. CIRP Ann 66(1):1–4. https://doi.org/10.1016/j.cirp.2017.04.095

    Article  Google Scholar 

  24. Charalambous G, Fletcher SR, Webb P (2017) The development of a Human Factors Readiness Level tool for implementing industrial human-robot collaboration. Int J Adv Manuf Technol 91(5-8):2465–2475. https://doi.org/10.1007/s00170-016-9876-6

    Article  Google Scholar 

  25. Chen M, Nikolaidis S, Soh H, Hsu D, Srinivasa S (2018) Planning with trust for human-robot collaboration. In: Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction. pp 307-315

  26. Tan JTC, Duan F, Kato R, Arai T, Hall E (2010) Collaboration planning by task analysis in human-robot collaborative manufacturing system. In: Hall E (ed) Advances in Robot Manipulators. IntechOpen, London

    Google Scholar 

  27. Fiore M, Clodic A, Alami R (2016) On planning and task achievement modalities for human-robot collaboration. Experimental Robotics. Springer, In, pp 293–306

    Google Scholar 

  28. Hawkins KP, Vo N, Bansal S, Bobick AF (2013) Probabilistic human action prediction and wait-sensitive planning for responsive human-robot collaboration. In: 13th IEEE-RAS International Conference on Humanoid Robots (Humanoids). pp 499-506

  29. Hayes B, Scassellati B (2016) Autonomously constructing hierarchical task networks for planning and human-robot collaboration. In: IEEE International Conference on Robotics and Automation (ICRA). pp 5469-5476

  30. Johannsmeier L, Haddadin S (2016) A hierarchical human-robot interaction-planning framework for task allocation in collaborative industrial assembly processes. IEEE Rob Autom Lett 2(1):41–48. https://doi.org/10.1109/LRA.2016.2535907

    Article  Google Scholar 

  31. Kardos C, Kovács A, Váncza J (2016) Towards feature-based human-robot assembly process planning. Procedia CIRP 57:516–521. https://doi.org/10.1016/j.procir.2016.11.089

    Article  Google Scholar 

  32. Ranz F, Hummel V, Sihn W (2017) Capability-based task allocation in human-robot collaboration. Procedia Manuf 9:182–189. https://doi.org/10.1016/j.promfg.2017.04.011

    Article  Google Scholar 

  33. Wang XV, Kemény Z, Váncza J, Wang L (2017) Human–robot collaborative assembly in cyber-physical production: Classification framework and implementation. CIRP Ann 66(1):5–8. https://doi.org/10.1016/j.cirp.2017.04.101

    Article  Google Scholar 

  34. Mateus JC, Claeys D, Limere V, Cottyn J, Aghezzaf E (2019) A structured methodology for the design of a human-robot collaborative assembly workplace. Int J Adv Manuf Technol 102(5-8):2663–2681. https://doi.org/10.1007/s00170-019-03356-3

    Article  Google Scholar 

  35. Kumar A, Dimitrakopoulos R, Maulen M (2020) Adaptive self-learning mechanisms for updating short-term production decisions in an industrial mining complex. J Intell Manuf 31(7):1795–1811. https://doi.org/10.1007/s10845-020-01562-5

    Article  Google Scholar 

  36. Xu W, Tang Q, Liu J, Liu Z, Zhou Z, Duc Truong P (2020) Disassembly sequence planning using discrete bees algorithm for human-robot collaboration in remanufacturing. Robot Comput Integr Manuf 62:101860. https://doi.org/10.1016/j.rcim.2019.101860

    Article  Google Scholar 

  37. Wang P, Liu H, Wang L, Gao RX (2018) Deep learning-based human motion recognition for predictive context-aware human-robot collaboration. CIRP Ann 67(1):17–20. https://doi.org/10.1016/j.cirp.2018.04.066

    Article  Google Scholar 

  38. Kartoun U, Stern H, Edan Y (2010) A human-robot collaborative reinforcement learning algorithm. J Intell Robot Syst 60(2):217–239. https://doi.org/10.1007/s10846-010-9422-y

    Article  MATH  Google Scholar 

  39. Akkaladevi SC, Plasch M, Maddukuri S, Eitzinger C, Pichler A, Rinner B (2018) Toward an interactive reinforcement based learning framework for human robot collaborative assembly processes. Front Rob AI 5. https://doi.org/10.3389/frobt.2018.00126

  40. Wang W, Li R, Chen Y, Diekel ZM, Jia Y (2019) Facilitating human-robot collaborative tasks by teaching-learning-collaboration from human demonstrations. IEEE Trans Autom Sci Eng 16(2):640–653. https://doi.org/10.1109/tase.2018.2840345

    Article  Google Scholar 

  41. Modares H, Ranatunga I, Lewis FL, Popa DO (2015) Optimized assistive human–robot interaction using reinforcement learning. IEEE Trans Cybern 46(3):655–667. https://doi.org/10.1109/TCYB.2015.2412554

    Article  Google Scholar 

  42. Gu S, Holly E, Lillicrap T, Levine S (2017) Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In: IEEE international conference on robotics and automation (ICRA). pp 3389-3396

  43. Ghadirzadeh A, Butepage J, Maki A, Kragic D, Bjorkman M (2016) A sensorimotor reinforcement learning framework for physical human-robot interaction. 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems.

  44. Chen C, Liu Y, Kreiss S, Alahi A (2019) Crowd-robot interaction: Crowd-aware robot navigation with attention-based deep reinforcement learning. In: 2019 International Conference on Robotics and Automation (ICRA). pp 6015-6022

  45. Trivedi M, Doshi P (2018) Inverse learning of robot behavior for collaborative planning. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp 1-9

  46. Foerster JN (2018) Deep multi-agent reinforcement learning. University of Oxford, Dissertation

    Google Scholar 

  47. Sutton RS, Barto AG (2018) Reinforcement learning: An introduction. MIT press, Cambridge

    MATH  Google Scholar 

Download references

Code availability

Code is available when required.

Funding

This work is supported by National Natural Science Foundation of China (Grant Nos. 51775399 and 51675389), the Fundamental Research Funds for the Central Universities (WUT: 2020III047), and the KTH-CSC programme of China Scholarship Council and KTH Royal Institute of Technology (No. 201906950003).

Author information

Authors and Affiliations

  1. School of Information Engineering, Wuhan University of Technology, Wuhan, 430070, China

    Zhihao Liu, Quan Liu, Wenjun Xu & Zude Zhou

  2. Hubei Key Laboratory of Broadband Wireless Communication and Sensor Networks, Wuhan University of Technology, Wuhan, 430070, China

    Zhihao Liu, Quan Liu & Wenjun Xu

  3. Department of Production Engineering, KTH Royal Institute of Technology, Stockholm, 10044, Sweden

    Zhihao Liu & Lihui Wang

Authors
  1. Zhihao Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Quan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lihui Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wenjun Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zude Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wenjun Xu.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, Z., Liu, Q., Wang, L. et al. Task-level decision-making for dynamic and stochastic human-robot collaboration based on dual agents deep reinforcement learning. Int J Adv Manuf Technol 115, 3533–3552 (2021). https://doi.org/10.1007/s00170-021-07265-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-021-07265-2

Keywords

Search

Navigation