Abstract
Many researchers have studied various visual communication cues (e.g. pointer, sketch, and hand gesture) in Mixed Reality remote collaboration systems for real-world tasks. However, the effect of combining them has not been so well explored. We studied the effect of these cues in four combinations: hand only, hand + pointer, hand + sketch, and hand + pointer + sketch, within the two user studies when a dependent view and independent view are supported respectively. In the first user study with the dependent view, the results showed that the hand gesture cue was the main visual communication cue and adding sketch cue to the hand gesture cue helped participants complete the task faster. In the second study with the independent view, the results showed that the hand gesture had an issue of local worker understanding remote expert’s hand gesture cue and the main visual communication cue was the pointer cue with fast completion time and high level of co-presence.
Similar content being viewed by others
References
Milgram P, Kishino F (1994) IEICE Trans Inf Syst 77(12):1321
Kim S, Lee G, Sakata N, Billinghurst M (2014) 2014 IEEE international symposium on mixed and augmented reality (ISMAR) pp. 83–92. https://doi.org/10.1109/ISMAR.2014.6948412
Kim S, Lee G, Huang W, Kim H, Woo W, Billinghurst M (2019) Proceedings of the 2019 CHI conference on human factors in computing systems (2019), pp 1–13
Lee G, Kim S, Lee Y, Dey A, Piumsomboon T, Norman M, Billinghurst M (2017) Adjunct proceedings of the 2017 IEEE international symposium on mixed and augmented reality, ISMAR-adjunct 2017 (Institute of Electrical and Electronics Engineers Inc., 2017), pp 79–80
Kim S, Lee GA, Sakata N (2013) 2013 IEEE international symposium on mixed and augmented reality (ISMAR), pp. 1–6. https://doi.org/10.1109/ISMAR.2013.6671833
Kim S, Lee GA, Ha S, Sakata N, Billinghurst M (2015) Proceedings of the 33rd annual ACM conference extended abstracts on human factors in computing systems (ACM, New York, NY, USA, 2015), CHI EA ’15, pp 1669–1674. 10.1145/2702613.2732838
Kim S, Billinghurst M, Lee G (2018) Comput Support Cooper Work (CSCW) 27(3):569. https://doi.org/10.1007/s10606-018-9324-2
Kasahara S, Rekimoto J (2014) Proceedings of the 5th augmented human international conference (ACM, New York, NY, USA, 2014), AH ’14, pp 46:1–46:8. http://doi.org/10.1145/2582051.2582097
Gauglitz S, Nuernberger B, Turk M, Höllerer T (2014) Proceedings of the 27th annual ACM symposium on user interface software and technology (ACM, New York, NY, USA, 2014), UIST ’14, pp 449–459. http://doi.org/10.1145/2642918.2647372
Huang W, Alem L, Tecchia F, Duh HBL (2018) J Multimod User Interfaces 12(2):77. https://doi.org/10.1007/s12193-017-0250-2
Kuzuoka H, Ishimoda G, Nishimura Y, Suzuki R, Kondo K (1995) Proceedings of the fourth european conference on computer-supported cooperative work ECSCW’95 (Springer, 1995), pp 181–196
Sakata N, Kurata T, Kato T, Kourogi M, Kuzuoka H (2003) ISWC, vol 2003 (Citeseer, 2003), vol 2003, p 7th
Gupta K, Lee GA, Billinghurst M (2016) IEEE Trans Vis Comput Graph 22(11):2413. https://doi.org/10.1109/TVCG.2016.2593778
Fussell SR, Setlock LD, Yang J, Ou J, Mauer E, Kramer ADI (2004) Hum-Comput Interaction 19(3):273. https://doi.org/10.1207/s15327051hci1903_3
Kim S, Lee GA, Sakata N, Daijnser A, Vartiainen E, Billinghurst, M (2013) IEEE international symposium on mixed and augmented reality (ISMAR) (2013), pp 261–262. http://doi.org/10.1109/ISMAR.2013.6671795
Gauglitz S, Nuernberger B, Turk M, Höllerer T (2014) Proceedings of the 20th ACM symposium on virtual reality software and technology (2014), pp 197–205
Huang W, Kim S, Billinghurst M, Alem L (2019) J Vis Commun Image Represent 58:428
Higuch K, Yonetani R, Sato Y (2016) Proceedings of the 2016 CHI conference on human factors in computing systems (ACM, New York, NY, USA, 2016), CHI ’16, pp 5180–5190. http://doi.org/10.1145/2858036.2858438
Fakourfar O, Ta K, Tang R, Bateman S, Tang A (2016) Proceedings of the 2016 CHI conference on human factors in computing systems (ACM, New York, NY, USA, 2016), CHI ’16, pp 1548–1560. http://doi.org/10.1145/2858036.2858171
Tang JC, Minneman SL (1990) Proceedings of the SIGCHI conference on human factors in computing systems (ACM, New York, NY, USA, 1990), CHI ’90, pp 313–320. http://doi.org/10.1145/97243.97302
Kato H, Billinghurst M (1999) Proceedings 2nd IEEE and ACM international workshop on augmented reality (IWAR’99) pp. 85–94. https://doi.org/10.1109/IWAR.1999.803809
Kim S, Billinghurst M, Lee C, Lee G (2018) KSII Trans Internet Inf Syst 12(12):6034. https://doi.org/10.3837/tiis.2018.12.023
Alem L, Tecchia F, Huang W (2011) (University of Wollongong, 2011), 11th underground coal operators’ conference
Adcock M, Anderson S, Thomas B (2013) Proceedings of the 12th ACM SIGGRAPH international conference on virtual-reality continuum and its applications in industry (2013), pp 235–242
Kirk D, Stanton Fraser D (2006) Proceedings of the SIGCHI conference on human factors in computing systems (ACM, New York, NY, USA, 2006), CHI ’06, pp 1191–1200. http://doi.org/10.1145/1124772.1124951
Kirk D, Rodden T, Fraser DS (2007) Proceedings of the SIGCHI conference on human factors in computing systems (ACM, New York, NY, USA, 2007), CHI ’07, pp 1039–1048. http://doi.org/10.1145/1240624.1240782
Friedman H (1979) Skill in nonverbal communication pp 2–27
Goldin-Meadow S (1999) Trends Cogni Sci 3(11):419. https://doi.org/10.1016/S1364-6613(99)01397-2
Teo T, Lee GA, Billinghurst M, Adcock M (2018) Proceedings of the 30th australian conference on computer-human interaction (2018), pp 406–410
Chen S, Chen M, Kunz A, Yantaç AE, Bergmark M, Sundin A, Fjeld M (2013) Proceedings of the 4th augmented human international conference (ACM, New York, NY, USA, 2013), AH ’13, pp 69–76. http://doi.org/10.1145/2459236.2459249
Stefik M, Bobrow DG, Foster G, Lanning S, Tatar D (1987) ACM Trans Inf Syst (TOIS) 5(2):147
Gutwin C, Greenberg S (1998) Proceedings of the 1998 ACM conference on computer supported cooperative work (1998), pp 207–216
Young J, Langlotz T, Cook M, Mills S, Regenbrecht H (2019) IEEE Trans Vis Comput Graph 25(5):1908
Teo T, Lawrence L, Lee GA, Billinghurst M, Adcock M (2019) Proceedings of the 2019 CHI conference on human factors in computing systems (2019), pp 1–14
Lee G, Teo THL, Kim S, Billinghurst M (2018) Proceedings of the IEEE international symposium for mixed and augmented reality, pp 153–164. http://doi.org/10.1109/ISMAR.2018.00051
Lee GA, Teo T, Kim S, Billinghurst M (2017) SIGGRAPH Asia 2017 Mobile graphics & interactive applications, pp 1–4
Piumsomboon T, Lee GA, Hart JD, Ens B, Lindeman RW, Thomas BH, Billinghurst M (2018) Proceedings of the 2018 CHI conference on human factors in computing systems (Association for Computing Machinery, Montreal QC, Canada, 2018), CHI ’18, pp. 1–13. https://doi.org/10.1145/3173574.3173620
Piumsomboon T, Day A, Ens B, Lee Y, Lee GA, Billinghurst M (2017) SA ’17. http://doi.org/10.1145/3132787.3139200
Gao L, Bai H, Piumsomboon T, Lee G, Lindeman RW, Billinghurst M (2017)
Meta2 website. https://www.metavision.com/
Durrant-Whyte H, Bailey T (2006) IEEE Robot Autom Mag 13(2):99
Fove head mounted display website. https://www.getfove.com/
Leap motion hand tracking website. https://www.leapmotion.com/
Weichert F, Bachmann D, Rudak B, Fisseler D (2013) Sensors 13(5):6380
Leap motion pinch example. https://gallery.leapmotion.com/pinch-draw/
Zijlstra FRH (1993) Efficiency in work behavior. A design approach for modern tools (Delft University of Technology, Delft, The Netherlands, 1993). Doctoral dissertation from Delft University of Technology
Zijlstra FRH, Doorn LV (1985) Delft, Netherlands p 43
Harms PC, Biocca PF (2004)
Wobbrock JO, Findlater L, Gergle D, Higgins JJ (2011) Proceedings of the SIGCHI conference on human factors in computing systems (ACM, New York, NY, USA), CHI ’11, pp 143–146. http://doi.org/10.1145/1978942.1978963
Meta2 unity. https://docs.metavision.com/external/doc/latest/unity_sdk_release_notes_260.html
Ricoh theta v. https://theta360.com/en/about/theta/v.html
Htc vive. https://www.vive.com/au/
T. Teo, A. F. Hayati, G. A. Lee, M. Billinghurst, M. Adcock, in 25th ACM Symposium on Virtual Reality Software and Technology (2019), pp. 1–11
Di Stasi LL, McCamy MB, Catena A, Macknik SL, Canas JJ, Martinez-Conde S (2013) European Journal of Neuroscience 38(3):2389
Siegenthaler E, Costela FM, McCamy MB, Di Stasi LL, Otero-Millan J, Sonderegger A, Groner R, Macknik S, Martinez-Conde S (2014) European Journal of Neuroscience 39(2):287
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Kim, S., Lee, G., Billinghurst, M. et al. The combination of visual communication cues in mixed reality remote collaboration. J Multimodal User Interfaces 14, 321–335 (2020). https://doi.org/10.1007/s12193-020-00335-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12193-020-00335-x