Skip to main content
SpringerLink
Log in

Real-time 3D video-based MR remote collaboration using gesture cues and virtual replicas

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

With the rapid development of mixed reality (MR) technology, many compact, lightweight, and powerful devices suitable for remote collaboration, such as MR headsets, hand trackers, and 3D cameras, become readily available, providing hardware and software support for remote collaboration. Consequently, exploring MR technologies for remote collaboration on physical industry tasks is becoming increasingly worthwhile. In many complex production scenarios, such as assembly tasks, significant gains can be achieved by having remote experts assist local workers to manipulate objects in local workspaces. However, it can be challenging for a remote expert to carry out effective spatial reference and action demonstration in a local scene. Sharing 3D stereoscopic scenes can provide depth perception and support remote experts to move and explore a local user’s environment freely. Previous studies have demonstrated that gesture-based interaction is natural and intuitive, and interaction based on virtual replicas can provide clear guidance, especially for industrial physical tasks. In this study, we develop an MR remote collaboration system that shares the stereoscopic scene of the local workspace by using real-time 3D video. This system combines gesture cues and virtual replicas in a complementary manner to support the remote expert to create augmented reality (AR) guidance for the local worker naturally and intuitively in the virtual reality immersive space. A formal user study was performed to explore the effects of two different modalities interface in industrial assembly tasks: our novel method of using the combination of virtual replicas and gesture cues in the 3D video (VG3DV), and a method similar to the popular method currently of using gesture cues in the 3D video (G3DV). We found that using the VG3DV can significantly improve the performance and user experience of MR remote collaboration in industrial assembly tasks. Finally, some conclusions and future research directions were given.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

Notes

  1. https://www.microsoft.com/zh-cn/hololens

  2. https://www.vive.com/cn/product/

  3. https://www.wampserver.com/

  4. https://github.com/topics/mixedrealitytoolkit-unity

  5. https://www.intelrealsense.com/developers/

  6. https://pointclouds.org/

  7. https://www.ultraleap.com/product/leap-motion-controller/

  8. https://www.intelrealsense.com/depth-camera-d435i/

  9. https://www.intel.com/content/www/us/en/download/645988/intel-realsense-d400-series-dynamic-calibration-tool.html?

  10. https://developer.vuforia.com/

  11. https://webrtc.org

References

  1. Villanueva A, Zhu Z, Liu Z, Peppler K, Redick T, Ramani K (2020) Meta-AR-App: an authoring platform for collaborative augmented reality in STEM classrooms. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems 1–14. https://doi.org/10.1145/3313831.3376146

  2. Oda O, Elvezio C, Sukan M, Feiner S, Tversky B (2015) Virtual replicas for remote assistance in virtual and augmented reality. Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology 405–415. https://doi.org/10.1145/2807442.2807497

  3. Wang P, Bai X, Billinghurst M, Zhang S, Wei S, Xu G, He W, Zhang X, Zhang J (2020) 3DGAM: using 3D gesture and CAD models for training on mixed reality remote collaboration. Multimed Tools Appl 80(20):31059–31084. https://doi.org/10.1007/s11042-020-09731-7

    Article  Google Scholar 

  4. Elvezio C, Sukan M, Feiner S, Tversky B (2015) Interactive visualizations for monoscopic eyewear to assist in manually orienting objects in 3D. IEEE Int Symp Mix Augment Real 2015:180–181. https://doi.org/10.1109/ISMAR.2015.54

    Article  Google Scholar 

  5. Wang P, Zhang S, Billinghurst M, Bai X, He W, Wang S, Sun M, Zhang S (2019) A comprehensive survey of AR/MR-based co-design in manufacturing. Eng Comp 36:1715–1738. https://doi.org/10.1007/s00366-019-00792-3

    Article  Google Scholar 

  6. Talmy L (2000) Semantic conflict and resolution. Toward a cognitive semantics, vol 2. MIT, Cambridge, pp 323–337

    Google Scholar 

  7. Heiser J, Tversky B, Silverman M (2004) Sketches for and from collaboration. Vis Spat Reason Des III 3:69–78

    Google Scholar 

  8. Piumsomboon T, Dey A, Ens B, Lee G, Billinghurst M (2017) CoVAR: mixed-platform remote collaborative augmented and virtual realities system with shared collaboration cues. IEEE Int Symp Mix Augment Real 2017:218–219. https://doi.org/10.1109/ISMAR-Adjunct.2017.72

    Article  Google Scholar 

  9. Teo T, Lawrence L, Lee G, Billinghurst M, Adcock M (2019) Mixed reality remote collaboration combining 360 video and 3D reconstruction. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems 1–14. https://doi.org/10.1145/3290605.3300431

  10. Venerella J, Franklin T, Sherpa L, Tang H, Zhu Z (2019) Integrating AR and VR for mobile remote collaboration. IEEE Int Symp Mix Augment Real Adjunct 2019:104–108

    Google Scholar 

  11. Anton D, Kurillo G, Bajcsy R (2018) User experience and interaction performance in 2D/3D telecollaboration. Future Generation Computer Systems. Futur Gener Comput Syst 82:77–88. https://doi.org/10.1016/j.future.2017.12.055

    Article  Google Scholar 

  12. Zillner J, Mendez E, Wagner D (2018) Augmented reality remote collaboration with dense reconstruction. IEEE Int Symp Mix Augment Real Adjunct 2018:18–19. https://doi.org/10.1109/ISMAR-Adjunct.2018.00028

    Article  Google Scholar 

  13. Piumsomboon T, Day A, Ens B, Lee Y, Lee G, Billinghurst M (2017) Exploring enhancements for remote mixed reality collaboration. SIGGRAPH Asia 2017 Mobile Graph Interact Appl 1–5. https://doi.org/10.1145/3132787.3139200

  14. Huang W, Alem L, Tecchia F, Duh HBL (2018) Augmented 3D hands: a gesture-based mixed reality system for distributed collaboration. J Multimodal User Interfaces 12(2):77–89. https://doi.org/10.1007/s12193-017-0250-2

    Article  Google Scholar 

  15. Yang P, Kitahara I, Ohta Y (2015) Remote mixed reality system supporting interactions with virtualized objects. ISMAR. https://doi.org/10.1109/ISMAR.2015.22

    Article  Google Scholar 

  16. Takeda Y, Matsuda A, Rekimoto J (2020) 3D human reconstruction from an image for mobile telepresence systems. 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops 2020:772–773. https://doi.org/10.1109/VRW50115.2020.00237

  17. Huang R, Zhang S, Bai X, Xu C, Huang B (2015) An effective subpart retrieval approach of 3D CAD models for manufacturing process reuse. Comput Ind 67:38–53. https://doi.org/10.1016/j.compind.2014.12.001

    Article  Google Scholar 

  18. Zhang Y, Zhang S, Huang R, Huang B, Yang L, Liang J (2021) A deep learning-based approach for machining process route generation. Int J Adv Manuf Technol. https://doi.org/10.1007/s00170-021-07412-9

    Article  Google Scholar 

  19. Kritzler M, Murr M, Michahelles F (2016) Remotebob: support of on-site workers via a telepresence remote expert system. Proceedings of the 6th International Conference on the Internet of Things 7–14. https://doi.org/10.1145/2991561.2991571

  20. Elvezio C, Sukan M, Oda O, Feiner S, Tversky B (2017) Remote collaboration in AR and VR using virtual replicas. ACM SIGGRAPH 2017 VR Village 1–2. https://doi.org/10.1145/3089269.3089281

  21. Stork S, Schuboe A (2010) Human cognition in manual assembly: theories and applications. Adv Eng Inform 24(3):320–328. https://doi.org/10.1016/j.aei.2010.05.010

    Article  Google Scholar 

  22. Makris S, Karagiannis P, Koukas S, Matthaiakis AS (2016) Augmented reality system for operator support in human–robot collaborative assembly. CIRP Ann 65(1):61–64. https://doi.org/10.1016/j.cirp.2016.04.038

    Article  Google Scholar 

  23. Deshpande A, Kim I (2018) The effects of augmented reality on improving spatial problem solving for object assembly. Adv Eng Inform 38:760–775. https://doi.org/10.1016/j.aei.2018.10.004

    Article  Google Scholar 

  24. Rios H, Hincapié M, Caponio A, Mercado E, Mendívil GE (2011) Augmented reality: an advantageous option for complex training and maintenance operations in aeronautic related processes. Int Conf Virtual Mix Real 2011:87–96. https://doi.org/10.1007/978-3-642-22021-0_11

    Article  Google Scholar 

  25. Ojer M, Alvarez H, Serrano I, Saiz FA, Barandiaran I, Aguinaga D, Querejeta L, Alejandro D (2020) Projection-based augmented reality assistance for manual electronic component assembly processes. Appl Sci 10(3):796. https://doi.org/10.3390/app10030796

    Article  Google Scholar 

  26. De Amicis R, Ceruti A, Francia D, Frizziero L, Simões B (2018) Augmented Reality for virtual user manual. Int J Interact Des Manuf 12(2):689–697. https://doi.org/10.1007/s12008-017-0451-7

    Article  Google Scholar 

  27. Funk M, Lischke L, Mayer S, Shirazi SA, Schmidt A (2018) Teach me how! Interactive assembly instructions using demonstration and in-situ projection. Assist Augment 2018:49–73. https://doi.org/10.1007/978-981-10-6404-3_4

    Article  Google Scholar 

  28. Neb A, Strieg F (2018) Generation of AR-enhanced assembly instructions based on assembly features. Procedia CIRP 72:1118–1123. https://doi.org/10.1016/j.procir.2018.03.210

    Article  Google Scholar 

  29. Bhattacharya B, Winer EH (2019) Augmented reality via expert demonstration authoring (AREDA). Comput Ind 105:61–79. https://doi.org/10.1016/j.compind.2018.04.021

    Article  Google Scholar 

  30. Su Y, Rambach J, Minaskan N, Lesur P, Pagani A (2019) Stricker D (2019) Deep multi-state object pose estimation for augmented reality assembly. IEEE Int Symp Mix Augment Real Adjunct 2019:222–227. https://doi.org/10.1109/ISMAR-Adjunct.2019.00-42

    Article  Google Scholar 

  31. Chang MML, Nee AYC, Ong SK (2020) Interactive AR-assisted product disassembly sequence planning (ARDIS). Int J Prod Res 58(16):4916–4931. https://doi.org/10.1080/00207543.2020.1730462

    Article  Google Scholar 

  32. Dimitropoulos N, Togias T, Zacharaki N, Michalos G, Makris S (2021) Seamless human–robot collaborative assembly using artificial intelligence and wearable devices. Appl Sci 11(12):5699. https://doi.org/10.3390/app11125699

    Article  Google Scholar 

  33. Dimitropoulos N, Togias T, Michalos G, Makris S (2021) Operator support in human–robot collaborative environments using AI enhanced wearable devices. Procedia Cirp 97:464–469. https://doi.org/10.1016/j.procir.2020.07.006

    Article  Google Scholar 

  34. Adcock M, Anderson S, Thomas B (2013) RemoteFusion: real time depth camera fusion for remote collaboration on physical tasks. Proceedings of the 12th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry 235–242. https://doi.org/10.1145/2534329.2534331

  35. Dai A, Nießner M, Zollhöfer M, Lzadi S, Theobalt C (2017) Bundlefusion: real-time globally consistent 3d reconstruction using on-the-fly surface reintegration. ACM Transact Graph 36(4):1–18. https://doi.org/10.1145/3054739

    Article  Google Scholar 

  36. Izadi S, Newcombe R, Kim D, Hilliges O, Molyneaux D, Hodges S, Kohli P, Shotton J, Davison A, Fitzgibbon A (2011) KinectFusion: real-time dynamic 3D surface reconstruction and interaction. ACM SIGGRAPH 2011 Talks 1–1. https://doi.org/10.1145/2037826.2037857

  37. Orts-Escolano S, Rhemann C, Fanello S, Chang W, Kowdle A, Degtyarev Y, Kim D, Davidson P, Khamis S, Dou M, Tankovich V, Loop C, Cai Q, Chou P, Mennicken S, Valentin J, Pradeep V, Wang S, Kang B, Kohli P, Lutchyn Y, Keskin C, Lzadi S (2016) Holoportation: virtual 3D teleportation in real-time. Proceedings of the 29th Annual Symposium on User Interface Software and Technology 741–754. https://doi.org/10.1145/2984511.2984517

  38. Prisacariu VA, Kähler O, Cheng MM, Ren CY, Valentin J, Torr PH, Reid ID, Murray DW (2014) A framework for the volumetric integration of depth images. 1-18 arXiv preprint http://arxiv.org/abs/1410.0925

  39. Nuernberger B, Turk M, Höllerer T (2017) Evaluating snapping-to-photos virtual travel interfaces for 3D reconstructed visual reality. Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology 1–11. https://doi.org/10.1145/3139131.3139138

  40. Wang P, Zhang S, Bai X, Billinghurst M, He W, Sun M, Chen Y, Lv H, Ji H (2019) 2.5DHANDS: a gesture-based MR remote collaborative platform. Int J Adv Manuf Technol 102(5):1339–1353. https://doi.org/10.1007/s00170-018-03237-1

    Article  Google Scholar 

  41. Le Chénéchal M, Duval T, Gouranton V, Royan J, Arnaldi B (2016) Vishnu: virtual immersive support for helping users an interaction paradigm for collaborative remote guiding in mixed reality. IEEE Third VR Int Workshop Collab Virtual Environ 2016:9–12. https://doi.org/10.1109/3DCVE.2016.7563559

    Article  Google Scholar 

  42. Wang P, Bai X, Billinghurst M, Zhang S, He W, Han D, Wang Y, Min H, Lan W, Han S (2020) Using a head pointer or eye gaze: the effect of gaze on spatial AR remote collaboration for physical tasks. Interact Comput 32:153–169. https://doi.org/10.1093/iwcomp/iwaa012

    Article  Google Scholar 

  43. Wang P, Zhang S, Bai X, Billinghurst M, Zhang L, Wang S, Han D, Lv H, Yan Y (2019) A gesture-and head-based multimodal interaction platform for MR remote collaboration. Int J Adv Manuf Technol 105(7):3031–3043. https://doi.org/10.1007/s00170-019-04434-2

    Article  Google Scholar 

  44. Al-Khafajiy M, Baker T, Chalmers C, Asim M, Kolivand H, Fahim M, Waraich A (2019) Remote health monitoring of elderly through wearable sensors. Multimed Tools Appl 78(17):24681–24706. https://doi.org/10.1007/s11042-018-7134-7

    Article  Google Scholar 

  45. García-Pereira I, Portalés C, Gimeno J, Casas S (2020) A collaborative augmented reality annotation tool for the inspection of prefabricated buildings. Multimed Tools Appl 79:6483–6501. https://doi.org/10.1007/s11042-019-08419-x

    Article  Google Scholar 

  46. Adcock M, Ranatunga D, Smith R, Thomas B (2014) Object-based touch manipulation for remote guidance of physical tasks. Proceedings of the 2nd ACM Symposium on Spatial User Interaction 113–122. https://doi.org/10.1145/2659766.2659768

  47. Tait M, Billinghurst M (2014) View independence in remote collaboration using AR. ISMAR. 309–310.

  48. Pierce J, Stearns B, Pausch R (1999) Voodoo dolls: seamless interaction at multiple scales in virtual environments. Proceedings of the 1999 Symposium on Interactive 3D Graphics 141–145.

  49. Wang P, Bai X, Billinghurst M, Zhang S, Han D, Lv H, He W, Yan Y, Zhang X, Min H (2019) An MR remote collaborative platform based on 3D CAD models for training in industry. 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) 91–92. https://doi.org/10.1109/ISMAR-Adjunct.2019.00038

  50. Fakourfar O, Ta K, Tang R, Bateman S, Tang A (2016) Stabilized annotations for mobile remote assistance. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems 1548–1560. https://doi.org/10.1145/2858036.2858171

  51. Kim S, Lee G, Huang W, Kim H, Woo W, Billinghurst M (2019) Evaluating the combination of visual communication cues for HMD-based mixed reality remote collaboration. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems 1–13. https://doi.org/10.1145/3290605.3300403

  52. Wang P, Bai X, Billinghurst M, Zhang S, Han D, Sun M, Wang Z, Han D, Sun M, Wang Z, Lv H, Han S (2020) Haptic feedback helps me? A VR-SAR remote collaborative system with tangible interaction. Int J Hum Comput Interact 36(13):1242–1257. https://doi.org/10.1080/10447318.2020.1732140

    Article  Google Scholar 

  53. Gao L, Bai H, He W, Billinghurst M, Lindeman R (2018) Real-time visual representations for mobile mixed reality remote collaboration. SIGGRAPH Asia 2018 Virtual & Augmented Reality 1–2. https://doi.org/10.1145/3275495.3275515

  54. Liu M, Tuzel O, Veeraraghavan A, Chellappa R (2010) Fast directional chamfer matching. IEEE Comput Soc Conf Comput Vis Pattern Recognit 2010:1696–1703. https://doi.org/10.1109/CVPR.2010.5539837

    Article  Google Scholar 

  55. Brooke J (1996) SUS-a quick and dirty usability scale. Usability Eval Ind 189(194):4–7

    Google Scholar 

  56. Hart SG, Staveland LE (1988) Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. Advances in Psychology, Vol. 52. Elsevier, North-Holland, pp 139–183

  57. Harms C, Biocca F (2004) Internal consistency and reliability of the networked minds measure of social presence

  58. De Pace F, Manuri F, Sanna A, Zappia D (2019) A comparison between two different approaches for a collaborative mixed-virtual environment in industrial maintenance. Front Robot AI 6(18):1–14. https://doi.org/10.3389/frobt.2019.00018

    Article  Google Scholar 

Download references

Acknowledgements

We would like to appreciate Professor Shusheng Zhang, Weiping He, and Xiaoliang Bai for their science leadership, and the constructive opinions of Dr. Peng Wang and Dr. Zhuo Wang significantly improved the paper. We also would like to appreciate Yuxiang Yan for donating the vise model used in our research. In addition, we would like to thank Yuxiang Yan and Quan Yu for helping experiment with data collection and preparing some supplementary materials.

Funding

This work was partly supported by Defense Industrial Technology Development Program (grant number XXXX2018213A001 and No. XXXX2018205B021), National Key R&D Program of China (grant number 2019YFB1703800, 2021YFB1714900, 2021YFB1716200, and 2020YFB1712503), the Programme of Introducing Talents of Discipline to Universities (111 Project), China (grant number B13044), and the Fundamental Research Funds for the Central Universities, NPU (grant number 3102020gxb003).

Author information

Authors and Affiliations

  1. Cyber-Physical Interaction Lab, Northwestern Polytechnical University, Xi’an, 710072, China

    Xiangyu Zhang, Xiaoliang Bai, Shusheng Zhang, Weiping He, Peng Wang, Yuxiang Yan & Quan Yu

  2. Cyber-Reality Innovation Centre (China), Nanjing, China

    Xiangyu Zhang, Xiaoliang Bai, Weiping He, Peng Wang, Yuxiang Yan & Quan Yu

  3. School of Mechanical Engineering, University of Shanghai for Science and Technology, Shanghai, China

    Zhuo Wang

Authors
  1. Xiangyu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoliang Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shusheng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Weiping He
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Peng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zhuo Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yuxiang Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Quan Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection, and analysis were performed by Xiangyu Zhang, Yuxiang Yan, and Quan Yu. The first draft of the manuscript was written by Xiangyu Zhang and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding authors

Correspondence to Xiaoliang Bai or Shusheng Zhang.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, X., Bai, X., Zhang, S. et al. Real-time 3D video-based MR remote collaboration using gesture cues and virtual replicas. Int J Adv Manuf Technol 121, 7697–7719 (2022). https://doi.org/10.1007/s00170-022-09654-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-022-09654-7

Keywords

Search

Navigation