Skip to main content
SpringerLink
Log in

Physics-based cooperative robotic digital twin framework for contactless delivery motion planning

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

Collaborative tasks in multiple systems have attracted considerable attention in contemporary manufacturing environments. Collaborative robots are representative agents for various collaborative industrial tasks. This study focuses on contactless delivery by considering multiple agents. A contactless delivery operation must be performed with well-synchronized cooperation among the sender, a receiver, and flying dynamics of the deliverable. The most challenging task is to predict the catching point of a thrown package by considering both collaborative robots. Accurate catching-point prediction and relevant operations were achieved using the proposed physics-informed neural network-based hybrid multi-stream deep learning framework. Various cooperative environments were considered using multi-modal input data and manufacturing conditions. The proposed framework incorporates these constraints, and the prediction is guided by flying dynamics and trajectory data. The effectiveness of the proposed framework was demonstrated by implementing multi-agent and related manufacturing environments in a digital twin system. The effectiveness of the proposed framework was proven experimentally using analyses and comparisons with existing machine learning methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability

Not applicable.

Code availability

Not applicable.

References

  1. Schmatz F, Beub F, Sender J, Flugge W (2020) Use of human-robot collaboration to enhance process monitoring of mechanical joining. Procedia Manuf 52:272–276. https://doi.org/10.1016/j.promfg.2020.11.045

    Article  Google Scholar 

  2. Zhang S, Li S, Li X, Xiong Y, Xie Z (2022) A human-robot dynamic fusion safety algorithm for collaborative operations of cobots. J Intell Robot Syst 104:18. https://doi.org/10.1007/s10846-021-01534-8

    Article  Google Scholar 

  3. He Y, Hu Y, Zhang P, Zhao B, Qi X, Zhang J (2019) Human-robot cooperative control based on virtual fixture in robot-assisted endoscopic sinus surgery. App Sci 9:1–21. https://doi.org/10.3390/app9081659

    Article  Google Scholar 

  4. Colan J, Nakanishi J, Aoyama T, Hasegawa Y (2020) A cooperative human-robot interface for constrained manipulation in robot-assisted endonasal surgery. App Sci 10:4809. https://doi.org/10.3390/app10144809

    Article  Google Scholar 

  5. Arai H, Takubo T, Hayashibara Y, Tanie K (2000) Human-robot cooperative manipulation using a virtual nonholonomic constraint. Proc 2020 IEEE CRA:4063–4069. https://doi.org/10.1109/ROBOT.2000.845365

  6. Li M, Ishii TRH (2007) Spatial motion constraints using virtual fixtures generated by anatomy. IEEE Trans Robot 23:4–19. https://doi.org/10.1109/TRO.2006.886838

    Article  Google Scholar 

  7. Mitterberger D, Atanasova L, Dorfler K, Gramazio F, Kohler M (2022) Tie a knot: human-robot cooperative workflow for assembling wooden structures using rope joints. Constr Robot 6:277–292. https://doi.org/10.1007/s41693-022-00083-2

    Article  Google Scholar 

  8. Biton A, Shoval S, Lerman Y (2022) The use of cobots for disabled and older adults. IFAC PapersOnLIne 55:96–101. https://doi.org/10.1016/j.ifacol.2022.04.176

    Article  Google Scholar 

  9. Su L, Shi L, Yu Y (2009) Collaborative assembly operation between two modular robots based on the optical position feedback. J Robot 2009:1–9. https://doi.org/10.1155/2009/214154

    Article  Google Scholar 

  10. Cherubini A, Navarro-Alarcon D (2021) Sensor-based control for collaborative robots: fundamentals, challengers, and opportunities. Front. Neurorobot. 14:1–14. https://doi.org/10.3389/fnbot.2020.576846

    Article  Google Scholar 

  11. Javaid M, Haleem A, Singh RP, Rab S, Suman R (2022) Significant applications of cobots in the field of manufacturing. Cog Robot 2:222–233. https://doi.org/10.1016/j.cogr.2022.10.001

    Article  Google Scholar 

  12. Lee H, Kim SD, Amin MAUA (2022) Control framework for collaborative robot using imitation learning-based teleoperation from human digital twin to robot digital twin. Mechatronics 85:102833. https://doi.org/10.1016/j.mechatronics.2022.102833

    Article  Google Scholar 

  13. Brito T, Queiroz J, Piardi L, Fernandes LA, Lima J, Leitao P (2020) A machine learning approach for collaborative robot smart manufacturing inspection for quality control systems. Proc Manuf 51:11–18. https://doi.org/10.1016/j.promfg.2020.10.003

    Article  Google Scholar 

  14. Ghadirzadeh A, Chen X, Yin W, Yi Z, Bjorkman M, Kragic D, Bjorkman M, Kragic D (2020) Human-centered collaborative robots with deep reinforcement learning. IEEE Robot Autom Lett 6:566–571. https://doi.org/10.1109/LRA.2020.3047730

    Article  Google Scholar 

  15. Huang Y, Silverio J, Rozo L, Caldwell DG (2018) Generalized task-parameterized skill learning. Proc IEEE ICRA 2018:5667–5674. https://doi.org/10.1109/ICRA.2018.8461079

    Article  Google Scholar 

  16. Shukla D, Erkent O, Piater J (2018) Learning semantics of gestural instructions for human-robot collaboration. Frontiers in Neurorobotics 12:7. https://doi.org/10.3389/fnbot.2018.00007

    Article  Google Scholar 

  17. Zhang J, Liu H, Chang Q, Wang L, Gao RX (2020) Recurrent neural network for motion trajectory prediction in human-robot collaborative assembly. CIRP Annals-Manufacturing Technology 69:9–12. https://doi.org/10.1016/j.cirp.2020.04.077

    Article  Google Scholar 

  18. Semeraro F, Griffiths A, Cangelosi A (2023) Human-robot collaboration and machine learning: a systematic review of recent research. Robot and CIM 79:102432. https://doi.org/10.1016/j.rcim.2022.102432

    Article  Google Scholar 

  19. Maeda GJ, Neumann G, Ewerton M, Lioutikov R, Kroemer O, Peters J (2017) Probabilistic movement primitives for coordination of multiple human–robot collaborative tasks. Auton. Robot 41:593–612. https://doi.org/10.1007/s10514-016-9556-2

    Article  Google Scholar 

  20. Qadeer N, Shah JH, Sharif M, Khan MA, Muhammad G, Zhang Y (2022) Intelligent tracking of mechanically thrown objects by industrial catching robot for automated in-plant logistics 4.0. Sensors 22:2113. https://doi.org/10.3390/s22062113

    Article  Google Scholar 

  21. Gayanov R, Mironov K, Mukhametshin R, Vokhmintsev A, Kurennov D (2018) Transportation of small objects by robotic throwing and catching: applying genetic programming for trajectory estimation. IFAC-PapersOnLine 51:533–537. https://doi.org/10.1016/j.ifacol.2018.11.271

    Article  Google Scholar 

  22. Zeng A, Song S, Lee J, Rodriguez A, Funkhouser T (2019) TossingBot: learning to throw arbitrary objects with residual physics. IEEE Transactions on Robot 36:1307–1319. https://doi.org/10.1109/TRO.2020.2988642

    Article  Google Scholar 

  23. Kim J, Lee H (2020) Adaptive human-machine evaluation framework using stochastic gradient descent-based reinforcement learning for dynamic competing network. App Sci 10:2558. https://doi.org/10.3390/app10072558

    Article  Google Scholar 

  24. Kim J, Lee H (2020) Cooperative multi-agent interaction and evaluation framework considering competitive networks with dynamic topology changes. App Sci 10:5828. https://doi.org/10.3390/app10175828

    Article  Google Scholar 

  25. Raissi M, Perdikaris P, Karniadakis GE (2019) Physics-informed neural networks: a deep leaning framework for solving forward and inverse problems involving nonlinear partial differential equations. J Computational Physics 378:686–707. https://doi.org/10.1016/j.jcp.2018.10.045

    Article  MathSciNet  MATH  Google Scholar 

  26. Cuomo S, Cola VSD, Giampaolo F, Rozza G, Raissi M, Piccialli F (2022) Scientific machine learning through physics-informed neural networks: where we are and what’s next. J Sci Comput 92:88. https://doi.org/10.1007/s10915-022-01939-z

    Article  MathSciNet  MATH  Google Scholar 

  27. Universal Robots (2016) Universal robots user manual UR10/CB3, Version 3.3.3. https://www.universal-robots.com/;2016 [accessed 12 March 2023]

  28. Copot C, Muresan C, Lonescu CM, Vanlanduit S, Keyser RD (2018) Calibration of UR10 robot controller through simple auto-turning approach. Robotics 7:1–20. https://doi.org/10.3390/robotics7030035

    Article  Google Scholar 

  29. Saif S, Rahmadani F, Lee H (2019) Implementation and simulation of cyber physical system for robotic arm control in smart factory. J KIIS 29:308–315. https://doi.org/10.5391/JKIIS.2019.29.4.308

    Article  Google Scholar 

  30. Lee H (2020) Development of real-time sketch-based on-the-spot process modeling and analysis system. J Manuf Sys 54:215–226. https://doi.org/10.1016/j.jmsy.2019.12.006

    Article  Google Scholar 

  31. Lee H (2019) Real-time manufacturing modeling and simulation framework using augmented reality and stochastic network analysis. Virtual Reality 23:85–99. https://doi.org/10.1007/s10055-018-0343-6

    Article  Google Scholar 

  32. Kuipers JB (2002) Quaternions and rotation sequences, 1st edn. Princeton University Press, Princeton, New Jersey

    MATH  Google Scholar 

  33. Asfahl CR (1992) Robots and manufacturing automation, 2nd edn. Wiley, New York

    Google Scholar 

  34. Song J (2022) Design and control of robot arms, 1st edn. Gyomoon, Paju, Gyeonggi

    Google Scholar 

  35. Toschi F, Sega M (2019) Flowing matter, 1st edn. SpringerOpen, Switzerland

    Book  Google Scholar 

  36. Bakht A, Nawaz A, Lee M, Lee H (2022) Ingredient analysis of biological wastewater using hybrid multi-stream deep learning framework. Comp and Chem Eng 168:108038. https://doi.org/10.1016/j.compchemeng.2022.108038

    Article  Google Scholar 

  37. Lee H, An H, Lee D (2022) Time-staged photoelastic image prediction using multi-stage convolutional autoencoders. J Eng App AI 116:105265. https://doi.org/10.1016/j.engappai.2022.105265

    Article  Google Scholar 

  38. Curry GL, Feldman RM (2011) Manufacturing systems modeling and analysis, 2nd edn. Springer Verlag, New York

    Book  MATH  Google Scholar 

Download references

Funding

This research study was supported by the Basic Science Research Program through National Research Foundation of Korea (NRF), funded by the Ministry of Education, S. Korea (grant number: NRF-2021R1A2C1008647).

Author information

Authors and Affiliations

  1. School of Industrial Engineering, Kumoh National Institute of Technology, Gumi, PO39177, South Korea

    Hyunsoo Lee

Authors
  1. Hyunsoo Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Hyunsoo Lee: methodology, performing experiments, writing, formal analysis, and funding acquisition.

Corresponding author

Correspondence to Hyunsoo Lee.

Ethics declarations

Ethical approval

Not applicable.

Consent to participate

The author declares that they all consent to participate in this research.

Consent for publication

The author declares that they all consent to publish the manuscript.

Competing interests

The author declares no competing interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lee, H. Physics-based cooperative robotic digital twin framework for contactless delivery motion planning. Int J Adv Manuf Technol 128, 1255–1270 (2023). https://doi.org/10.1007/s00170-023-11956-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-023-11956-3

Keywords

Search

Navigation