Skip to main content
SpringerLink
Log in

A gesture- and head-based multimodal interaction platform for MR remote collaboration

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

In this paper, we present a projector-based mixed reality (MR) remote collaborative system which enables remote users to collaboratively work on a physical task using gesture and head pointing (GHP). Using this platform, we studied the effects of GHP in a typical manufacturing use case. Our system supports natural and intuitive multimodal interaction based on GHP, and it can project the remote user’s GHP into the local environment to enhance remote collaboration. Our prototype system was compared with an augmented reality (AR) condition (ANNOTATION), which is the most popular method currently for AR/MR remote collaboration. We found a significant difference between the ANNOTATION and GHP conditions in terms of performance. The GHP system significantly improved the collaborative experience (e.g., awareness of the user’s attention), empathy (e.g., co-presence), and remote interaction. Moreover, we discuss the implications of this research and directions for future research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. https://www.leapmotion.com

  2. https://developer.microsoft.com/en-us/windows/kinect

  3. https://www.intelrealsense.com

  4. https://www.baslerweb.com

  5. https://unity3d.com/cn/unity

  6. https://measuringu.com/sus/

References

  1. Wang P, Zhang S, Billinghurst M, Bai X, He W, Wang S, Sun M, Zhang X (2019) A comprehensive survey of AR/MR-based co-design in manufacturing. Eng Comput:1–24

  2. Nee AYC, Ong SK, Chryssolouris G, Mourtzis D (2012) Augmented reality applications in design and manufacturing. CIRP Ann Manuf Technol 61(2):657–679

    Article  Google Scholar 

  3. Ong SK, Yuan ML, Nee AYC (2008) Augmented reality applications in manufacturing: a survey. Int J Prod Res 46(10):2707–2742

    Article  Google Scholar 

  4. Zubizarreta J, Aguinaga I, Amundarain A (2019) A framework for augmented reality guidance in industry. Int J Adv Manuf Technol 102(9–12):4095–4108

    Article  Google Scholar 

  5. Wang Y, Zhang S, Wan B, He W, Bai X (2018) Point cloud and visual feature-based tracking method for an augmented reality-aided mechanical assembly system. Int J Adv Manuf Technol 99(9–12):2341–2352

    Article  Google Scholar 

  6. Fussell SR, Setlock LD, Kraut RE (2003) Effects of head-mounted and scene-oriented video systems on remote collaboration on physical tasks. Conference on Human Factors in Computing Systems. ACM 513–520

  7. Ranjan A, Birnholtz JP, Balakrishnan R (2007) Dynamic shared visual spaces: experimenting with automatic camera control in a remote repair task. Sigchi Conference on Human Factors in Computing Systems. ACM 1177–1186

  8. Anton D, Kurillo G, Bajcsy R (2017) User experience and interaction performance in 2D/3D telecollaboration. Futur Gener Comput Syst 82:77–88

    Article  Google Scholar 

  9. Anton D, Kurillo G, Yang AY, Bajcsy R (2017) Augmented telemedicine platform for real-time remote medical consultation. Multimedia modeling. Springer International Publishing, pp 77–89

  10. Gurevich P, Lanir J, Cohen B (2015) Design and implementation of TeleAdvisor: a projection-based augmented reality system for remote collaboration. Comput Supported Coop Work 24(6):527–562

    Article  Google Scholar 

  11. Fakourfar O, Ta K, Tang R, Bateman S, Tang A (2016) Stabilized annotations for mobile remote assistance. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM 1548–1560

  12. D’Angelo S, Gergle D (2018) An eye for design: gaze visualizations for remote collaborative work. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (p.349). ACM

  13. Otsuki M, Maruyama K, Kuzuoka H, SUZUKI Y. (2018) Effects of enhanced gaze presentation on gaze leading in remote collaborative physical tasks. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, p 368

  14. Lee G, Kim S, Lee Y, Dey A (2017) [POSTER] Mutually shared gaze in augmented video conference. IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE:79–80

  15. Gupta K, Lee GA, Billinghurst M (2016) Do you see what I see? The effect of gaze tracking on task space remote collaboration. IEEE Trans Vis Comput Graph 22(11):2413–2422

    Article  Google Scholar 

  16. Wang P, Zhang S, Bai X, Billinghurst M, He W, Zhang L, Du J, Wang S (2018) [POSTER] Do you know what I mean? An MR-based collaborative platform. IEEE International Symposium on Mixed and Augmented Reality (ISMAR). https://www.ismar2018.org/papers/ismar2018_poster_1044.html

  17. Gao L, Bai H, Lindeman R, Billinghurst M. (2017) Static local environment capturing and sharing for MR remote collaboration. In: SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications. ACM, p 17

  18. Andrist S, Gleicher M, Mutlu B (2017) Looking coordinated: bidirectional gaze mechanisms for collaborative interaction with virtual characters. CHI Conference on Human Factors in Computing Systems. ACM 2571–2582

  19. Wang P, Zhang S, Bai X, Billinghurst M, He W, Wang S, Zhang X, Du J, Chen Y (2019) Head pointer or eye gaze: which helps more in MR remote collaboration. IEEE VR 2019. http://ieeevr.org/2019/program/posters.html

  20. Huang W, Alem L, Tecchia F, Duh HB (2017) Augmented 3D hands: a gesture-based mixed reality system for distributed collaboration. J Multimodal User Interfaces 2:1–13

    Google Scholar 

  21. Wang S, Parsons M, Stonemclean J, Rogers P, Boyd S, Hoover K, Meruvia-Pastor O, Gong M, Smith A (2017) Augmented reality as a telemedicine platform for remote procedural training. Sensors 17(10):2294

    Article  Google Scholar 

  22. O’Neill J, Castellani S, Roulland F, Juliano C, Dai L, Roulland F, Hairon N (2011) From ethnographic study to mixed reality: a remote collaborative troubleshooting system. ACM 2011 Conference on Computer Supported Cooperative Work. ACM (CSCW), p 225–234

  23. Higuchi K, Yonetani R, Sato Y (2016) Can eye help you? Effects of visualizing eye fixations on remote collaboration scenarios for physical tasks. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, p 5180–5190

  24. Akkil D, James JM, Isokoski P, Kangas J (2016) GazeTorch: enabling gaze awareness in collaborative physical tasks. In: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, p 1151–1158

  25. Kytö M, Ens B, Piumsomboon T, Lee GA, Billinghurst M (2018) Pinpointing: precise head-and eye-based target selection for augmented reality. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, p 81

  26. Kraut RE, Fussell SR, Siegel J (2003) Visual information as a conversational resource in collaborative physical tasks. Hum-Comput Interact 18(1–2):13–49

    Article  Google Scholar 

  27. Fussell SR, Setlock LD, Parker EM, Yang J (2003) Assessing the value of a cursor pointing device for remote collaboration on physical tasks. Extended abstracts of the 2003 Conference on Human Factors in Computing Systems, CHI 2003, Ft. Lauderdale, Florida, USA, April. DBLP 788–789

  28. Ou J, Oh LM, Yang J, Fussell SR (2005) Effects of task properties, partner actions, and message content on eye gaze patterns in a collaborative task. In: Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, p 231–240

  29. Brennan SE, Chen X, Dickinson CA, Neider MB, Zelinsky GJ (2008) Coordinating cognition: The costs and benefits of shared gaze during collaborative search. Cognition 106(3):1465–1477

    Article  Google Scholar 

  30. Fussell SR, Setlock LD, Yang J, Ou J, Mauer E, Kramer ADI (2004) Gestures over video streams to support remote collaboration on physical tasks. Hum-Comput Interact 19(3):273–309

    Article  Google Scholar 

  31. Ou J, Chen X, Fussell SR, Yang J (2003) DOVE: drawing over video environment. Eleventh ACM International Conference on Multimedia. ACM 100–101

  32. Li J, Wessels A, Alem L, Stitzlein C (2007) Exploring interface with representation of gesture for remote collaboration. Ozchi:179–182

  33. Alem L, Li J (2011) A study of gestures in a video-mediated collaborative assembly task. Int J Hum-Comput Int 2011(3):1

    Google Scholar 

  34. Tecchia F, Alem L, Huang W (2012) 3D helping hands: a gesture based MR system for remote collaboration. ACM SIGGRAPH International Conference on Virtual-Reality Continuum and ITS Applications in Industry. ACM (VRCAI) 323–328

  35. Kirk D, Rodden T (2007) Turn it this way: grounding collaborative action with remote gestures. Conference on Human Factors in Computing Systems, CHI 2007, San Jose, California, USA, April 28 - May. DBLP 1039–1048

  36. Kirk D, Crabtree A, Rodden T (2005) Ways of the hands. ECSCW 2005. Springer, Netherlands, pp 1–21

    Book  Google Scholar 

  37. Kirk D, Fraser DS (2006) Comparing remote gesture technologies for supporting collaborative physical tasks. Conference on Human Factors in Computing Systems, CHI 2006, Montréal, Québec, Canada, April DBLP:1191–1200

  38. Huang W, Alem L (2013) HandsinAir: a wearable system for remote collaboration on physical tasks. Proceedings of the 2013 conference on Computer supported cooperative work companion. ACM (CSCW) 153–156

  39. Wang P, Zhang S, Bai X, Billinghurst M, He W, Sun M, Chen Y, Lv H, Ji H (2019) 2.5DHANDS: a gesture-based MR remote collaborative platform. Int J Adv Manuf Technol 120(5–8):1339–1353

    Article  Google Scholar 

  40. Deng S, Jiang N, Chang J, Guo S, Zhang JJ (2017) Understanding the impact of multimodal interaction using gaze informed mid-air gesture control in 3D virtual objects manipulation. Int J Hum Comput St 105:68–80

    Article  Google Scholar 

  41. Gurevich P, Lanir J, Cohen B, Cohen B, Stone R (2012) TeleAdvisor: a versatile augmented reality tool for remote assistance. Sigchi Conference on Human Factors in Computing Systems. ACM 619–622

  42. Tait M, Billinghurst M (2015) The effect of view independence in a collaborative AR system. Comput Supported Coop Work 24(6):563–589

    Article  Google Scholar 

  43. D’Angelo S, Gergle D (2016) Gazed and confused: understanding and designing shared gaze for remote collaboration. CHI Conference on Human Factors in Computing Systems. ACM

  44. Brooke J (1996) SUS -- a quick and dirty usability scale. Usability Evaluation in Industry

  45. Harms C, Biocca F (2004) Internal consistency and reliability of the networked minds measure of social presence. Alcanizm, Rey, Seventh International Workshop: Presence

Download references

Acknowledgments

We would like to thank Yuming Zheng for donating the water pump used in our research. We would also like to thank Professor Weiping He for science leadership, and Weiqi Lan and Haitao Min for the experiment data collection. Specifically, Mark Billinghurst carefully checked the English of an early version of the paper and helped the authors to significantly improve the paper.

Funding

This research was financially sponsored by the civil aircraft special project (MJZ-2017-G73) and Dongguan Science and Technology Equipment Project (KZ2018-05).

Author information

Authors and Affiliations

  1. Cyber-Physical Interaction Lab, Northwestern Polytechnical University, Xi’an, 710072, China

    Peng Wang, Shusheng Zhang, Xiaoliang Bai, Mark Billinghurst, Li Zhang, Shuxia Wang, Dechuan Han, Hao Lv & Yuxiang Yan

  2. Cyber-Reality Innovation Centre(China), Nanjing, China

    Peng Wang, Shusheng Zhang, Xiaoliang Bai, Mark Billinghurst, Li Zhang, Shuxia Wang, Dechuan Han & Yuxiang Yan

  3. Empathic Computing Lab, University of South Australia, Mawson Lakes, Australia

    Mark Billinghurst

Authors
  1. Peng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shusheng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaoliang Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mark Billinghurst
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Li Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shuxia Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Dechuan Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Hao Lv
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Yuxiang Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Peng Wang or Xiaoliang Bai.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

ESM 1

(AVI 48176 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, P., Zhang, S., Bai, X. et al. A gesture- and head-based multimodal interaction platform for MR remote collaboration. Int J Adv Manuf Technol 105, 3031–3043 (2019). https://doi.org/10.1007/s00170-019-04434-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-019-04434-2

Keywords

Search

Navigation