Skip to main content
SpringerLink
Log in

Exploring human-machine collaboration in industry: a systematic literature review of digital twin and robotics interfaced with extended reality technologies

  • Critical Review
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

This systematic literature review presents the latest advancements and insights about digital twin technology and robotics interfaced with extended reality in the context of Industry 4.0. As the extended reality technologies emerge, it results in an increasing overlap between digital twins and human-robot interactions in industrial settings, promoting collaboration between operators and cobots in manufacturing environments. The objective of this study is to serve as a valuable resource for researchers and practitioners working in the field of Industry 4.0. It aims to highlight the latest developments and innovations in the application of digital twins and robotics interfaced with extended reality technologies in manufacturing. By extracting data from relevant articles, it provides a comprehensive understanding of the current state-of-the-art in this field by: analyzing the favored extended reality interfaces for digital twin and robotics interactions; analyzing the digital twin and physical twin interaction; evaluating the digital twin application levels and pillars through extended reality interfacing; and introducing a new concept called augmented perception for creating new physical-digital interactions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. The interactions between physical twin and digital twin can take 4 different forms: digital model (DM), digital generator (DG), digital shadow (DS), and digital twin (DT).

  2. The total number of use cases is not equal to the number of selected articles: 30 papers have been included in this SLR which cover 46 use cases.

References

  1. Coupry C, Noblecourt S, Richard P et al (2021) BIM-Based Digital Twin and XR Devices to Improve Maintenance Procedures in Smart Buildings: A Literature Review. Applied Sciences 11:6810. https://doi.org/10.3390/app11156810

    Article  Google Scholar 

  2. Sjarov M, Lechler T, Fuchs J et al (2020) The Digital Twin Concept in Industry – A Review and Systematization. In: 2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA). pp 1789–1796. https://doi.org/10.1109/ETFA46521.2020.9212089

  3. Perno M, Hvam L, Haug A (2022) Implementation of digital twins in the process industry: A systematic literature review of enablers and barriers. Computers in Industry 134:103558. https://doi.org/10.1016/j.compind.2021.103558

    Article  Google Scholar 

  4. Kaul R, Ossai C, Forkan ARM et al (2023) The role of AI for developing digital twins in healthcare: The case of cancer care. WIREs Data Mining and Knowledge Discovery 13:e1480. https://doi.org/10.1002/widm.1480

    Article  Google Scholar 

  5. Honghong S, Gang Y, Haijiang L et al (2023) Digital twin enhanced BIM to shape full life cycle digital transformation for bridge engineering. Automation in Construction 147:104736. https://doi.org/10.1016/j.autcon.2022.104736

    Article  Google Scholar 

  6. Wang H, Chen X, Jia F, Cheng X (2023) Digital twin-supported smart city: Status, challenges and future research directions. Expert Systems with Applications 217:119531. https://doi.org/10.1016/j.eswa.2023.119531

    Article  Google Scholar 

  7. Falcão R, Matar R, Rauch B (2023) Using I4.0 Digital Twins in Agriculture. In: Batista T, Bureš T, Raibulet C, Muccini H (eds) Software Architecture. ECSA 2022 Tracks and Workshops. Springer International Publishing, Cham, pp 483–498. https://doi.org/10.1007/978-3-031-36889-9_32

  8. Liu Z, Hansen DW, Chen Z (2023) Leveraging Digital Twins to Support Industrial Symbiosis Networks: A Case Study in the Norwegian Wood Supply Chain Collaboration. Sustainability 15:2647. https://doi.org/10.3390/su15032647

    Article  Google Scholar 

  9. Burghardt A, Szybicki D, Gierlak P et al (2020) Programming of Industrial Robots Using Virtual Reality and Digital Twins. Applied Sciences 10:486. https://doi.org/10.3390/app10020486

    Article  Google Scholar 

  10. Directorate-General for Research and Innovation (European Commission), Breque M, De Nul L, Petridis A (2021) Industry 5.0: towards a sustainable, human centric and resilient European industry. Publications Office of the European Union, LU

  11. Li C, Zheng P, Li S et al (2022) AR-assisted digital twin-enabled robot collaborative manufacturing system with human-in-the-loop. Robotics and Computer-Integrated Manufacturing 76:102321. https://doi.org/10.1016/j.rcim.2022.102321

    Article  Google Scholar 

  12. Batistute A, Santos E, Takieddine K et al (2021) Extended Reality for Teleoperated Mobile Robots. In: 2021 Latin American Robotics Symposium (LARS), 2021 Brazilian Symposium on Robotics (SBR), and 2021 Workshop on Robotics in Education (WRE). pp 19–24. https://doi.org/10.1109/LARS/SBR/WRE54079.2021.9605474

  13. Pérez L, Rodríguez-Jiménez S, Rodríguez N et al (2020) Digital Twin and Virtual Reality Based Methodology for Multi-Robot Manufacturing Cell Commissioning. Applied Sciences 10:3633. https://doi.org/10.3390/app10103633

    Article  Google Scholar 

  14. Dallel M, Havard V, Dupuis Y, Baudry D (2023) Digital twin of an industrial workstation: A novel method of an auto-labeled data generator using virtual reality for human action recognition in the context of human-robot collaboration. Engineering Applications of Artificial Intelligence 118:105655. https://doi.org/10.1016/j.engappai.2022.105655

    Article  Google Scholar 

  15. Qiu C, Zhou S, Liu Z et al (2019) Digital assembly technology based on augmented reality and digital twins: a review. Virtual Reality & Intelligent Hardware 1:597–610. https://doi.org/10.1016/j.vrih.2019.10.002

    Article  Google Scholar 

  16. Kitchenham B, Pearl Brereton O, Budgen D et al (2009) Systematic literature reviews in software engineering - A systematic literature review. Information and Software Technology 51:7–15. https://doi.org/10.1016/j.infsof.2008.09.009

    Article  Google Scholar 

  17. Assila A, Dhouib A, Monla Z, Zghal M (2022) Integration of Augmented, Virtual and Mixed Reality with Building Information Modeling: A Systematic Review. In: Chen JYC, Fragomeni G (eds) Virtual, Augmented and Mixed Reality: Design and Development. Springer International Publishing, Cham, pp 3–19. https://doi.org/10.1007/978-3-031-05939-1_1

  18. Suh IH, Lim GH, Hwang W et al (2007) Ontology-based multi-layered robot knowledge framework (OMRKF) for robot intelligence. In: 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems. pp 429–436. https://doi.org/10.1109/IROS.2007.4399082

  19. Kritzinger W, Karner M, Traar G et al (2018) Digital Twin in manufacturing: A categorical literature review and classification. IFAC-PapersOnLine 51:1016–1022. https://doi.org/10.1016/j.ifacol.2018.08.474

    Article  Google Scholar 

  20. Tekinerdogan B, Verdouw C (2020) Systems Architecture Design Pattern Catalog for Developing Digital Twins. Sensors 20:5103. https://doi.org/10.3390/s20185103

    Article  Google Scholar 

  21. Macci‘o S, Shaaban M, Carfì A et al (2023) RICO-MR: An Open-Source Architecture for Robot Intent Communication through Mixed Reality. https://doi.org/10.48550/arXiv.2309.04765

  22. Caiza G, Sanz R (2022) Digital Twin for Monitoring an Industrial Process Using Augmented Reality. In: 2022 17th Iberian Conference on Information Systems and Technologies (CISTI). pp 1–5. https://doi.org/10.23919/CISTI54924.2022.9820356

  23. Begout P, Kubicki S, Bricard E, Duval T (2022) Augmented Reality Authoring of Digital Twins: Design, Implementation and Evaluation in an Industry 4.0 Context. Frontiers in Virtual Reality 3:918685. https://doi.org/10.3389/frvir.2022.918685

  24. Vidal-Balea A, Blanco-Novoa O, Fraga-Lamas P et al (2022) A Collaborative Industrial Augmented Reality Digital Twin: Developing the Future of Shipyard 4.0. In: International Summit Smart City 360\(^{\circ }\). Springer, pp 104–120. https://doi.org/10.1007/978-3-031-06371-8_8

  25. Calandra D, Pratticó FG, Cannavó A et al (2022) Digital twin- and extended reality-based telepresence for collaborative robot programming in the 6G perspective. Digital Communications and Networks. https://doi.org/10.1016/j.dcan.2022.10.007

    Article  Google Scholar 

  26. Choi SH, Park K-B, Roh DH et al (2022) An integrated mixed reality system for safety-aware human-robot collaboration using deep learning and digital twin generation. Robotics and Computer-Integrated Manufacturing 73:102258. https://doi.org/10.1016/j.rcim.2021.102258

    Article  Google Scholar 

  27. Weistroffer V, Keith F, Bisiaux A et al (2022) Using Physics-Based Digital Twins and Extended Reality for the Safety and Ergonomics Evaluation of Cobotic Workstations. Front Virtual Real 3:781830. https://doi.org/10.3389/frvir.2022.781830

    Article  Google Scholar 

  28. Geng R, Li M, Hu Z et al (2022) Digital Twin in smart manufacturing: remote control and virtual machining using VR and AR technologies. Struct Multidisc Optim 65:321. https://doi.org/10.1007/s00158-022-03426-3

    Article  Google Scholar 

  29. Garg G, Kuts V, Anbarjafari G (2021) Digital Twin for FANUC Robots: Industrial Robot Programming and Simulation Using Virtual Reality. Sustainability 13:10336. https://doi.org/10.3390/su131810336

    Article  Google Scholar 

  30. Pizzagalli S, Kuts V, Bondarenko Y, Otto T (2021) Evaluation of Virtual Reality Interface Interaction Methods for Digital Twin Industrial Robot Programming and Control, A Pilot Study. In: IMECE2021. Volume 2B: Advanced Manufacturing. https://doi.org/10.1115/IMECE2021-69408

  31. Stavropoulos P, Papacharalampopoulos A, Siatras V, Mourtzis D (2021) An AR based Digital Twin for Laser based manufacturing process monitoring. In: Procedia CIRP. pp 258–263. https://doi.org/10.1016/j.procir.2021.09.044

  32. Müller M, Mielke J, Pavlovskyi Y et al (2021) Real-time combination of material flow simulation, digital twins of manufacturing cells, an AGV and a mixed-reality application. Procedia CIRP 104:1607–1612. https://doi.org/10.1016/j.procir.2021.11.271

    Article  Google Scholar 

  33. Aschenbrenner D, Rieder JSI, van Tol D et al (2020) Mirrorlabs - creating accessible Digital Twins of robotic production environment with Mixed Reality. In: 2020 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR). pp 43–48. https://doi.org/10.1109/AIVR50618.2020.00017

  34. Williams R (2020) Augmented reality assisted calibration of digital twins of mobile robots. In: IFAC-PapersOnLine. pp 203–208. https://doi.org/10.1016/j.ifacol.2020.11.033

  35. Kuts V, Otto T, Bondarenko Y, Yu F (2021) Digital Twin: Collaborative Virtual Reality Environment for Multi-Purpose Industrial Applications. American Society of Mechanical Engineers Digital Collection. https://doi.org/10.1115/IMECE2020-23390

    Article  Google Scholar 

  36. Cai Y, Wang Y, Burnett M (2020) Using augmented reality to build digital twin for reconfigurable additive manufacturing system. Journal of Manufacturing Systems 56:598–604. https://doi.org/10.1016/j.jmsy.2020.04.005

    Article  Google Scholar 

  37. Deac G-C, Georgescu C-N, Laurentiu Popa C, Emil Cotet C (2020) Virtual Reality Digital Twin for a Smart Factory. IJMO 190–195. https://doi.org/10.7763/IJMO.2020.V10.769

  38. Zhu Z, Liu C, Xu X (2019) Visualisation of the Digital Twin data in manufacturing by using Augmented Reality. Procedia CIRP 81:898–903. https://doi.org/10.1016/j.procir.2019.03.223

    Article  Google Scholar 

  39. Havard V, Jeanne B, Lacomblez M, Baudry D (2019) Digital twin and virtual reality: a co-simulation environment for design and assessment of industrial workstations. Production & Manufacturing Research 7:472–489. https://doi.org/10.1080/21693277.2019.1660283

    Article  Google Scholar 

  40. Kuts V, Otto T, Tähemaa T, Bondarenko Y (2019) Digital twin based synchronised control and simulation of the industrial robotic cell using virtual reality. Journal of Machine Engineering 19:128–144. https://doi.org/10.5604/01.3001.0013.0464

    Article  Google Scholar 

  41. Alfrink M (2019) Enhanced interaction with industrial robots through extended reality relying on simulation-based digital twins. In: Proceedings of the ISC. p 90

  42. Lacomblez M, Jeanne B, Havard V, Baudry D (2018) Co-Simulation Architecture Between a Digital Twin and a Virtual Reality Environment in an Industrial Context. In: Advances in Manufacturing Technology XXXII. IOS Press, pp 429–434. https://doi.org/10.3233/978-1-61499-902-7-429

  43. Kuts V, Otto T, Caldarola EG et al (2018) Enabling the teaching factory leveraging a virtual reality system based on the Digital Twin. In: Proceedings of the 15th Annual EuroVR Conference

  44. Eyre J, Freeman C (2018) Immersive applications of industrial digital twins. The Industrial Track of EuroVR 2018:11–13

    Google Scholar 

  45. Rabah S, Assila A, Khouri E et al (2018) Towards improving the future of manufacturing through digital twin and augmented reality technologies. Procedia Manufacturing 17:460–467. https://doi.org/10.1016/j.promfg.2018.10.070

    Article  Google Scholar 

  46. Perianes-Rodriguez A, Waltman L, van Eck NJ (2016) Constructing bibliometric networks: A comparison between full and fractional counting. Journal of Informetrics 10:1178–1195. https://doi.org/10.1016/j.joi.2016.10.006

    Article  Google Scholar 

  47. Mukherjee D, Gupta K, Chang LH, Najjaran H (2022) A Survey of Robot Learning Strategies for Human-Robot Collaboration in Industrial Settings. Robotics and Computer-Integrated Manufacturing 73:102231. https://doi.org/10.1016/j.rcim.2021.102231

    Article  Google Scholar 

  48. Rokhsaritalemi S, Sadeghi-Niaraki A, Choi S-M (2020) A Review on Mixed Reality: Current Trends. Challenges and Prospects. Applied Sciences 10:636. https://doi.org/10.3390/app10020636

  49. Tao F, Xiao B, Qi Q et al (2022) Digital twin modeling. Journal of Manufacturing Systems 64:372–389. https://doi.org/10.1016/j.jmsy.2022.06.015

    Article  Google Scholar 

  50. Lechler T, Fuchs J, Sjarov M et al (2020) Introduction of a comprehensive Structure Model for the Digital Twin in Manufacturing. In: 2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA). IEEE, Vienna, Austria, pp 1773–1780. https://doi.org/10.1109/ETFA46521.2020.9212030

  51. Kuts V, Cherezova N, Sarkans M, Otto T (2020) Digital Twin: Industrial robot kinematic model integration to the virtual reality environment. Journal of Machine Engineering 20:53–64

Download references

Acknowledgements

This work, carried out within the framework of the JENII project, benefited from a State grant managed by the National Research Agency under the France 2030 program, with the reference ANR-21-DMES-0006

Funding

This work benefited from a state grant managed by the National Research Agency under the France 2030 program, with the reference ANR-21-DMES-0006.

Author information

Authors and Affiliations

  1. CESI LINEACT, 80 avenue Edmund Halley Rouen Madrillet Innovation, 76800, Saint-Étienne-du-Rouvray, France

    Yassine Feddoul, Fabrice Duval, Vincent Havard & David Baudry

  2. HESAM Université, 15 Rue Soufflot, 75005, Paris, France

    Yassine Feddoul

  3. CESI LINEACT, 4 Pl. de Boston, Hérouville-Saint-Clair, 14200, France

    Nicolas Ragot

  4. CESI LINEACT, 7 bis Av. Robert Schuman, 51100, Reims, France

    Ahlem Assila

Authors
  1. Yassine Feddoul
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nicolas Ragot
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fabrice Duval
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Vincent Havard
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. David Baudry
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ahlem Assila
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to reviewing articles and extracting relevant information to report it to this manuscript.

Corresponding author

Correspondence to Yassine Feddoul.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

All authors are contributed equally to this work.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Feddoul, Y., Ragot, N., Duval, F. et al. Exploring human-machine collaboration in industry: a systematic literature review of digital twin and robotics interfaced with extended reality technologies. Int J Adv Manuf Technol 129, 1917–1932 (2023). https://doi.org/10.1007/s00170-023-12291-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-023-12291-3

Keywords

Search

Navigation