Skip to main content
Log in

The combination of visual communication cues in mixed reality remote collaboration

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Many researchers have studied various visual communication cues (e.g. pointer, sketch, and hand gesture) in Mixed Reality remote collaboration systems for real-world tasks. However, the effect of combining them has not been so well explored. We studied the effect of these cues in four combinations: hand only, hand + pointer, hand + sketch, and hand + pointer + sketch, within the two user studies when a dependent view and independent view are supported respectively. In the first user study with the dependent view, the results showed that the hand gesture cue was the main visual communication cue and adding sketch cue to the hand gesture cue helped participants complete the task faster. In the second study with the independent view, the results showed that the hand gesture had an issue of local worker understanding remote expert’s hand gesture cue and the main visual communication cue was the pointer cue with fast completion time and high level of co-presence.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Milgram P, Kishino F (1994) IEICE Trans Inf Syst 77(12):1321

    Google Scholar 

  2. Kim S, Lee G, Sakata N, Billinghurst M (2014) 2014 IEEE international symposium on mixed and augmented reality (ISMAR) pp. 83–92. https://doi.org/10.1109/ISMAR.2014.6948412

  3. Kim S, Lee G, Huang W, Kim H, Woo W, Billinghurst M (2019) Proceedings of the 2019 CHI conference on human factors in computing systems (2019), pp 1–13

  4. Lee G, Kim S, Lee Y, Dey A, Piumsomboon T, Norman M, Billinghurst M (2017) Adjunct proceedings of the 2017 IEEE international symposium on mixed and augmented reality, ISMAR-adjunct 2017 (Institute of Electrical and Electronics Engineers Inc., 2017), pp 79–80

  5. Kim S, Lee GA, Sakata N (2013) 2013 IEEE international symposium on mixed and augmented reality (ISMAR), pp. 1–6. https://doi.org/10.1109/ISMAR.2013.6671833

  6. Kim S, Lee GA, Ha S, Sakata N, Billinghurst M (2015) Proceedings of the 33rd annual ACM conference extended abstracts on human factors in computing systems (ACM, New York, NY, USA, 2015), CHI EA ’15, pp 1669–1674. 10.1145/2702613.2732838

  7. Kim S, Billinghurst M, Lee G (2018) Comput Support Cooper Work (CSCW) 27(3):569. https://doi.org/10.1007/s10606-018-9324-2

    Article  Google Scholar 

  8. Kasahara S, Rekimoto J (2014) Proceedings of the 5th augmented human international conference (ACM, New York, NY, USA, 2014), AH ’14, pp 46:1–46:8. http://doi.org/10.1145/2582051.2582097

  9. Gauglitz S, Nuernberger B, Turk M, Höllerer T (2014) Proceedings of the 27th annual ACM symposium on user interface software and technology (ACM, New York, NY, USA, 2014), UIST ’14, pp 449–459. http://doi.org/10.1145/2642918.2647372

  10. Huang W, Alem L, Tecchia F, Duh HBL (2018) J Multimod User Interfaces 12(2):77. https://doi.org/10.1007/s12193-017-0250-2

    Article  Google Scholar 

  11. Kuzuoka H, Ishimoda G, Nishimura Y, Suzuki R, Kondo K (1995) Proceedings of the fourth european conference on computer-supported cooperative work ECSCW’95 (Springer, 1995), pp 181–196

  12. Sakata N, Kurata T, Kato T, Kourogi M, Kuzuoka H (2003) ISWC, vol 2003 (Citeseer, 2003), vol 2003, p 7th

  13. Gupta K, Lee GA, Billinghurst M (2016) IEEE Trans Vis Comput Graph 22(11):2413. https://doi.org/10.1109/TVCG.2016.2593778

    Article  Google Scholar 

  14. Fussell SR, Setlock LD, Yang J, Ou J, Mauer E, Kramer ADI (2004) Hum-Comput Interaction 19(3):273. https://doi.org/10.1207/s15327051hci1903_3

    Article  Google Scholar 

  15. Kim S, Lee GA, Sakata N, Daijnser A, Vartiainen E, Billinghurst, M (2013) IEEE international symposium on mixed and augmented reality (ISMAR) (2013), pp 261–262. http://doi.org/10.1109/ISMAR.2013.6671795

  16. Gauglitz S, Nuernberger B, Turk M, Höllerer T (2014) Proceedings of the 20th ACM symposium on virtual reality software and technology (2014), pp 197–205

  17. Huang W, Kim S, Billinghurst M, Alem L (2019) J Vis Commun Image Represent 58:428

    Article  Google Scholar 

  18. Higuch K, Yonetani R, Sato Y (2016) Proceedings of the 2016 CHI conference on human factors in computing systems (ACM, New York, NY, USA, 2016), CHI ’16, pp 5180–5190. http://doi.org/10.1145/2858036.2858438

  19. Fakourfar O, Ta K, Tang R, Bateman S, Tang A (2016) Proceedings of the 2016 CHI conference on human factors in computing systems (ACM, New York, NY, USA, 2016), CHI ’16, pp 1548–1560. http://doi.org/10.1145/2858036.2858171

  20. Tang JC, Minneman SL (1990) Proceedings of the SIGCHI conference on human factors in computing systems (ACM, New York, NY, USA, 1990), CHI ’90, pp 313–320. http://doi.org/10.1145/97243.97302

  21. Kato H, Billinghurst M (1999) Proceedings 2nd IEEE and ACM international workshop on augmented reality (IWAR’99) pp. 85–94. https://doi.org/10.1109/IWAR.1999.803809

  22. Kim S, Billinghurst M, Lee C, Lee G (2018) KSII Trans Internet Inf Syst 12(12):6034. https://doi.org/10.3837/tiis.2018.12.023

    Article  Google Scholar 

  23. Alem L, Tecchia F, Huang W (2011) (University of Wollongong, 2011), 11th underground coal operators’ conference

  24. Adcock M, Anderson S, Thomas B (2013) Proceedings of the 12th ACM SIGGRAPH international conference on virtual-reality continuum and its applications in industry (2013), pp 235–242

  25. Kirk D, Stanton Fraser D (2006) Proceedings of the SIGCHI conference on human factors in computing systems (ACM, New York, NY, USA, 2006), CHI ’06, pp 1191–1200. http://doi.org/10.1145/1124772.1124951

  26. Kirk D, Rodden T, Fraser DS (2007) Proceedings of the SIGCHI conference on human factors in computing systems (ACM, New York, NY, USA, 2007), CHI ’07, pp 1039–1048. http://doi.org/10.1145/1240624.1240782

  27. Friedman H (1979) Skill in nonverbal communication pp 2–27

  28. Goldin-Meadow S (1999) Trends Cogni Sci 3(11):419. https://doi.org/10.1016/S1364-6613(99)01397-2

    Article  Google Scholar 

  29. Teo T, Lee GA, Billinghurst M, Adcock M (2018) Proceedings of the 30th australian conference on computer-human interaction (2018), pp 406–410

  30. Chen S, Chen M, Kunz A, Yantaç AE, Bergmark M, Sundin A, Fjeld M (2013) Proceedings of the 4th augmented human international conference (ACM, New York, NY, USA, 2013), AH ’13, pp 69–76. http://doi.org/10.1145/2459236.2459249

  31. Stefik M, Bobrow DG, Foster G, Lanning S, Tatar D (1987) ACM Trans Inf Syst (TOIS) 5(2):147

    Article  Google Scholar 

  32. Gutwin C, Greenberg S (1998) Proceedings of the 1998 ACM conference on computer supported cooperative work (1998), pp 207–216

  33. Young J, Langlotz T, Cook M, Mills S, Regenbrecht H (2019) IEEE Trans Vis Comput Graph 25(5):1908

    Article  Google Scholar 

  34. Teo T, Lawrence L, Lee GA, Billinghurst M, Adcock M (2019) Proceedings of the 2019 CHI conference on human factors in computing systems (2019), pp 1–14

  35. Lee G, Teo THL, Kim S, Billinghurst M (2018) Proceedings of the IEEE international symposium for mixed and augmented reality, pp 153–164. http://doi.org/10.1109/ISMAR.2018.00051

  36. Lee GA, Teo T, Kim S, Billinghurst M (2017) SIGGRAPH Asia 2017 Mobile graphics & interactive applications, pp 1–4

  37. Piumsomboon T, Lee GA, Hart JD, Ens B, Lindeman RW, Thomas BH, Billinghurst M (2018) Proceedings of the 2018 CHI conference on human factors in computing systems (Association for Computing Machinery, Montreal QC, Canada, 2018), CHI ’18, pp. 1–13. https://doi.org/10.1145/3173574.3173620

  38. Piumsomboon T, Day A, Ens B, Lee Y, Lee GA, Billinghurst M (2017) SA ’17. http://doi.org/10.1145/3132787.3139200

  39. Gao L, Bai H, Piumsomboon T, Lee G, Lindeman RW, Billinghurst M (2017)

  40. Meta2 website. https://www.metavision.com/

  41. Durrant-Whyte H, Bailey T (2006) IEEE Robot Autom Mag 13(2):99

    Article  Google Scholar 

  42. Fove head mounted display website. https://www.getfove.com/

  43. Leap motion hand tracking website. https://www.leapmotion.com/

  44. Weichert F, Bachmann D, Rudak B, Fisseler D (2013) Sensors 13(5):6380

    Article  Google Scholar 

  45. Leap motion pinch example. https://gallery.leapmotion.com/pinch-draw/

  46. Zijlstra FRH (1993) Efficiency in work behavior. A design approach for modern tools (Delft University of Technology, Delft, The Netherlands, 1993). Doctoral dissertation from Delft University of Technology

  47. Zijlstra FRH, Doorn LV (1985) Delft, Netherlands p 43

  48. Harms PC, Biocca PF (2004)

  49. Wobbrock JO, Findlater L, Gergle D, Higgins JJ (2011) Proceedings of the SIGCHI conference on human factors in computing systems (ACM, New York, NY, USA), CHI ’11, pp 143–146. http://doi.org/10.1145/1978942.1978963

  50. Meta2 unity. https://docs.metavision.com/external/doc/latest/unity_sdk_release_notes_260.html

  51. Ricoh theta v. https://theta360.com/en/about/theta/v.html

  52. Htc vive. https://www.vive.com/au/

  53. T. Teo, A. F. Hayati, G. A. Lee, M. Billinghurst, M. Adcock, in 25th ACM Symposium on Virtual Reality Software and Technology (2019), pp. 1–11

  54. Di Stasi LL, McCamy MB, Catena A, Macknik SL, Canas JJ, Martinez-Conde S (2013) European Journal of Neuroscience 38(3):2389

    Article  Google Scholar 

  55. Siegenthaler E, Costela FM, McCamy MB, Di Stasi LL, Otero-Millan J, Sonderegger A, Groner R, Macknik S, Martinez-Conde S (2014) European Journal of Neuroscience 39(2):287

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Seungwon Kim.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, S., Lee, G., Billinghurst, M. et al. The combination of visual communication cues in mixed reality remote collaboration. J Multimodal User Interfaces 14, 321–335 (2020). https://doi.org/10.1007/s12193-020-00335-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-020-00335-x

Keywords

Navigation