A gesture- and head-based multimodal interaction platform for MR remote collaboration

  • Peng WangEmail author
  • Shusheng Zhang
  • Xiaoliang BaiEmail author
  • Mark Billinghurst
  • Li Zhang
  • Shuxia Wang
  • Dechuan Han
  • Hao Lv
  • Yuxiang Yan


In this paper, we present a projector-based mixed reality (MR) remote collaborative system which enables remote users to collaboratively work on a physical task using gesture and head pointing (GHP). Using this platform, we studied the effects of GHP in a typical manufacturing use case. Our system supports natural and intuitive multimodal interaction based on GHP, and it can project the remote user’s GHP into the local environment to enhance remote collaboration. Our prototype system was compared with an augmented reality (AR) condition (ANNOTATION), which is the most popular method currently for AR/MR remote collaboration. We found a significant difference between the ANNOTATION and GHP conditions in terms of performance. The GHP system significantly improved the collaborative experience (e.g., awareness of the user’s attention), empathy (e.g., co-presence), and remote interaction. Moreover, we discuss the implications of this research and directions for future research.


Remote collaboration Multimodal interaction Gestures Head pointing Mixed reality 



We would like to thank Yuming Zheng for donating the water pump used in our research. We would also like to thank Professor Weiping He for science leadership, and Weiqi Lan and Haitao Min for the experiment data collection. Specifically, Mark Billinghurst carefully checked the English of an early version of the paper and helped the authors to significantly improve the paper.

Funding information

This research was financially sponsored by the civil aircraft special project (MJZ-2017-G73) and Dongguan Science and Technology Equipment Project (KZ2018-05).

Supplementary material

170_2019_4434_MOESM1_ESM.avi (47 mb)
ESM 1 (AVI 48176 kb)


  1. 1.
    Wang P, Zhang S, Billinghurst M, Bai X, He W, Wang S, Sun M, Zhang X (2019) A comprehensive survey of AR/MR-based co-design in manufacturing. Eng Comput:1–24Google Scholar
  2. 2.
    Nee AYC, Ong SK, Chryssolouris G, Mourtzis D (2012) Augmented reality applications in design and manufacturing. CIRP Ann Manuf Technol 61(2):657–679CrossRefGoogle Scholar
  3. 3.
    Ong SK, Yuan ML, Nee AYC (2008) Augmented reality applications in manufacturing: a survey. Int J Prod Res 46(10):2707–2742CrossRefGoogle Scholar
  4. 4.
    Zubizarreta J, Aguinaga I, Amundarain A (2019) A framework for augmented reality guidance in industry. Int J Adv Manuf Technol 102(9–12):4095–4108CrossRefGoogle Scholar
  5. 5.
    Wang Y, Zhang S, Wan B, He W, Bai X (2018) Point cloud and visual feature-based tracking method for an augmented reality-aided mechanical assembly system. Int J Adv Manuf Technol 99(9–12):2341–2352CrossRefGoogle Scholar
  6. 6.
    Fussell SR, Setlock LD, Kraut RE (2003) Effects of head-mounted and scene-oriented video systems on remote collaboration on physical tasks. Conference on Human Factors in Computing Systems. ACM 513–520Google Scholar
  7. 7.
    Ranjan A, Birnholtz JP, Balakrishnan R (2007) Dynamic shared visual spaces: experimenting with automatic camera control in a remote repair task. Sigchi Conference on Human Factors in Computing Systems. ACM 1177–1186Google Scholar
  8. 8.
    Anton D, Kurillo G, Bajcsy R (2017) User experience and interaction performance in 2D/3D telecollaboration. Futur Gener Comput Syst 82:77–88CrossRefGoogle Scholar
  9. 9.
    Anton D, Kurillo G, Yang AY, Bajcsy R (2017) Augmented telemedicine platform for real-time remote medical consultation. Multimedia modeling. Springer International Publishing, pp 77–89Google Scholar
  10. 10.
    Gurevich P, Lanir J, Cohen B (2015) Design and implementation of TeleAdvisor: a projection-based augmented reality system for remote collaboration. Comput Supported Coop Work 24(6):527–562CrossRefGoogle Scholar
  11. 11.
    Fakourfar O, Ta K, Tang R, Bateman S, Tang A (2016) Stabilized annotations for mobile remote assistance. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM 1548–1560Google Scholar
  12. 12.
    D’Angelo S, Gergle D (2018) An eye for design: gaze visualizations for remote collaborative work. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (p.349). ACMGoogle Scholar
  13. 13.
    Otsuki M, Maruyama K, Kuzuoka H, SUZUKI Y. (2018) Effects of enhanced gaze presentation on gaze leading in remote collaborative physical tasks. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, p 368Google Scholar
  14. 14.
    Lee G, Kim S, Lee Y, Dey A (2017) [POSTER] Mutually shared gaze in augmented video conference. IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE:79–80Google Scholar
  15. 15.
    Gupta K, Lee GA, Billinghurst M (2016) Do you see what I see? The effect of gaze tracking on task space remote collaboration. IEEE Trans Vis Comput Graph 22(11):2413–2422CrossRefGoogle Scholar
  16. 16.
    Wang P, Zhang S, Bai X, Billinghurst M, He W, Zhang L, Du J, Wang S (2018) [POSTER] Do you know what I mean? An MR-based collaborative platform. IEEE International Symposium on Mixed and Augmented Reality (ISMAR).
  17. 17.
    Gao L, Bai H, Lindeman R, Billinghurst M. (2017) Static local environment capturing and sharing for MR remote collaboration. In: SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications. ACM, p 17Google Scholar
  18. 18.
    Andrist S, Gleicher M, Mutlu B (2017) Looking coordinated: bidirectional gaze mechanisms for collaborative interaction with virtual characters. CHI Conference on Human Factors in Computing Systems. ACM 2571–2582Google Scholar
  19. 19.
    Wang P, Zhang S, Bai X, Billinghurst M, He W, Wang S, Zhang X, Du J, Chen Y (2019) Head pointer or eye gaze: which helps more in MR remote collaboration. IEEE VR 2019.
  20. 20.
    Huang W, Alem L, Tecchia F, Duh HB (2017) Augmented 3D hands: a gesture-based mixed reality system for distributed collaboration. J Multimodal User Interfaces 2:1–13Google Scholar
  21. 21.
    Wang S, Parsons M, Stonemclean J, Rogers P, Boyd S, Hoover K, Meruvia-Pastor O, Gong M, Smith A (2017) Augmented reality as a telemedicine platform for remote procedural training. Sensors 17(10):2294CrossRefGoogle Scholar
  22. 22.
    O’Neill J, Castellani S, Roulland F, Juliano C, Dai L, Roulland F, Hairon N (2011) From ethnographic study to mixed reality: a remote collaborative troubleshooting system. ACM 2011 Conference on Computer Supported Cooperative Work. ACM (CSCW), p 225–234Google Scholar
  23. 23.
    Higuchi K, Yonetani R, Sato Y (2016) Can eye help you? Effects of visualizing eye fixations on remote collaboration scenarios for physical tasks. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, p 5180–5190Google Scholar
  24. 24.
    Akkil D, James JM, Isokoski P, Kangas J (2016) GazeTorch: enabling gaze awareness in collaborative physical tasks. In: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, p 1151–1158Google Scholar
  25. 25.
    Kytö M, Ens B, Piumsomboon T, Lee GA, Billinghurst M (2018) Pinpointing: precise head-and eye-based target selection for augmented reality. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, p 81Google Scholar
  26. 26.
    Kraut RE, Fussell SR, Siegel J (2003) Visual information as a conversational resource in collaborative physical tasks. Hum-Comput Interact 18(1–2):13–49CrossRefGoogle Scholar
  27. 27.
    Fussell SR, Setlock LD, Parker EM, Yang J (2003) Assessing the value of a cursor pointing device for remote collaboration on physical tasks. Extended abstracts of the 2003 Conference on Human Factors in Computing Systems, CHI 2003, Ft. Lauderdale, Florida, USA, April. DBLP 788–789Google Scholar
  28. 28.
    Ou J, Oh LM, Yang J, Fussell SR (2005) Effects of task properties, partner actions, and message content on eye gaze patterns in a collaborative task. In: Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, p 231–240Google Scholar
  29. 29.
    Brennan SE, Chen X, Dickinson CA, Neider MB, Zelinsky GJ (2008) Coordinating cognition: The costs and benefits of shared gaze during collaborative search. Cognition 106(3):1465–1477CrossRefGoogle Scholar
  30. 30.
    Fussell SR, Setlock LD, Yang J, Ou J, Mauer E, Kramer ADI (2004) Gestures over video streams to support remote collaboration on physical tasks. Hum-Comput Interact 19(3):273–309CrossRefGoogle Scholar
  31. 31.
    Ou J, Chen X, Fussell SR, Yang J (2003) DOVE: drawing over video environment. Eleventh ACM International Conference on Multimedia. ACM 100–101Google Scholar
  32. 32.
    Li J, Wessels A, Alem L, Stitzlein C (2007) Exploring interface with representation of gesture for remote collaboration. Ozchi:179–182Google Scholar
  33. 33.
    Alem L, Li J (2011) A study of gestures in a video-mediated collaborative assembly task. Int J Hum-Comput Int 2011(3):1Google Scholar
  34. 34.
    Tecchia F, Alem L, Huang W (2012) 3D helping hands: a gesture based MR system for remote collaboration. ACM SIGGRAPH International Conference on Virtual-Reality Continuum and ITS Applications in Industry. ACM (VRCAI) 323–328Google Scholar
  35. 35.
    Kirk D, Rodden T (2007) Turn it this way: grounding collaborative action with remote gestures. Conference on Human Factors in Computing Systems, CHI 2007, San Jose, California, USA, April 28 - May. DBLP 1039–1048Google Scholar
  36. 36.
    Kirk D, Crabtree A, Rodden T (2005) Ways of the hands. ECSCW 2005. Springer, Netherlands, pp 1–21CrossRefGoogle Scholar
  37. 37.
    Kirk D, Fraser DS (2006) Comparing remote gesture technologies for supporting collaborative physical tasks. Conference on Human Factors in Computing Systems, CHI 2006, Montréal, Québec, Canada, April DBLP:1191–1200Google Scholar
  38. 38.
    Huang W, Alem L (2013) HandsinAir: a wearable system for remote collaboration on physical tasks. Proceedings of the 2013 conference on Computer supported cooperative work companion. ACM (CSCW) 153–156Google Scholar
  39. 39.
    Wang P, Zhang S, Bai X, Billinghurst M, He W, Sun M, Chen Y, Lv H, Ji H (2019) 2.5DHANDS: a gesture-based MR remote collaborative platform. Int J Adv Manuf Technol 120(5–8):1339–1353CrossRefGoogle Scholar
  40. 40.
    Deng S, Jiang N, Chang J, Guo S, Zhang JJ (2017) Understanding the impact of multimodal interaction using gaze informed mid-air gesture control in 3D virtual objects manipulation. Int J Hum Comput St 105:68–80CrossRefGoogle Scholar
  41. 41.
    Gurevich P, Lanir J, Cohen B, Cohen B, Stone R (2012) TeleAdvisor: a versatile augmented reality tool for remote assistance. Sigchi Conference on Human Factors in Computing Systems. ACM 619–622Google Scholar
  42. 42.
    Tait M, Billinghurst M (2015) The effect of view independence in a collaborative AR system. Comput Supported Coop Work 24(6):563–589CrossRefGoogle Scholar
  43. 43.
    D’Angelo S, Gergle D (2016) Gazed and confused: understanding and designing shared gaze for remote collaboration. CHI Conference on Human Factors in Computing Systems. ACMGoogle Scholar
  44. 44.
    Brooke J (1996) SUS -- a quick and dirty usability scale. Usability Evaluation in IndustryGoogle Scholar
  45. 45.
    Harms C, Biocca F (2004) Internal consistency and reliability of the networked minds measure of social presence. Alcanizm, Rey, Seventh International Workshop: PresenceGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2019

Authors and Affiliations

  • Peng Wang
    • 1
    • 2
    Email author
  • Shusheng Zhang
    • 1
    • 2
  • Xiaoliang Bai
    • 1
    • 2
    Email author
  • Mark Billinghurst
    • 1
    • 2
    • 3
  • Li Zhang
    • 1
    • 2
  • Shuxia Wang
    • 1
    • 2
  • Dechuan Han
    • 1
    • 2
  • Hao Lv
    • 1
  • Yuxiang Yan
    • 1
    • 2
  1. 1.Cyber-Physical Interaction LabNorthwestern Polytechnical UniversityXi’anChina
  2. 2.Cyber-Reality Innovation Centre(China)NanjingChina
  3. 3.Empathic Computing LabUniversity of South AustraliaMawson LakesAustralia

Personalised recommendations