Skip to main content
Log in

Exploring interaction techniques for 360 panoramas inside a 3D reconstructed scene for mixed reality remote collaboration

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Remote collaboration using mixed reality (MR) enables two separated workers to collaborate by sharing visual cues. A local worker can share his/her environment to the remote worker for a better contextual understanding. However, prior techniques were using either 360 video sharing or a complicated 3D reconstruction configuration. This limits the interactivity and practicality of the system. In this paper we show an interactive and easy-to-configure MR remote collaboration technique enabling a local worker to easily share his/her environment by integrating 360 panorama images into a low-cost 3D reconstructed scene as photo-bubbles and projective textures. This enables the remote worker to visit past scenes on either an immersive 360 panoramic scenery, or an interactive 3D environment. We developed a prototype and conducted a user study comparing the two modes of how 360 panorama images could be used in a remote collaboration system. Results suggested that both photo-bubbles and projective textures can provide high social presence, co-presence and low cognitive load for solving tasks while each have its advantage and limitations. For example, photo-bubbles are good for a quick navigation inside the 3D environment without depth perception while projective textures are good for spatial understanding but require physical efforts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Adcock M, Feng D, Thomas B (2013) Visualization of off-surface 3D viewpoint locations in spatial augmented reality. https://doi.org/10.1145/2491367.2491378

  2. Palmer D, Adcock M, Smith J, Hutchins M, Gunn C, Stevenson D, Taylor K (2007) Proceedings of the 2007 conference of the computer-human interaction special interest group (CHISIG) of Australia on Computer–human interaction: design: activities, artifacts and environments—OZCHI ’07. ACM Press, New York, p 103. https://doi.org/10.1145/1324892.1324911. http://portal.acm.org/citation.cfm?doid=1324892.1324911

  3. Orts-Escolano S, Rhemann C, Fanello S, Chang W, Kowdle A, Degtyarev Y, Kim D, Davidson PL, Khamis S, Dou M, Tankovich V, Loop C, Cai Q, Chou PA, Mennicken S, Valentin J, Pradeep V, Wang S, Kang SB, Kohli P, Lutchyn Y, Keskin C, Izadi S (2016) Holoportation: virtual 3D teleportation in real-time. https://doi.org/10.1145/2984511.2984517

  4. Pejsa T, Kantor J, Benko H, Ofek E, Wilson AD (2016) Room2Room: enabling life-size telepresence in a projected augmented reality environment. https://doi.org/10.1145/2818048.2819965

  5. Piumsomboon T, Lee GA, Hart JD, Ens B, Lindeman RW, Thomas BH, Billinghurst M (2018) Mini-Me. https://doi.org/10.1145/3173574.3173620

  6. Le Chenechal M, Chalme S, Duval T, Royan J, Gouranton V, Arnaldi B (2015) International conference on collaboration technologies and systems (CTS). IEEE, pp 233–240. https://doi.org/10.1109/CTS.2015.7210428. http://ieeexplore.ieee.org/document/7210428/

  7. Garcia-Pereira I, Gimeno J, Perez M, Portales C, Casas S (2018) IEEE international symposium on mixed and augmented reality adjunct (ISMAR-Adjunct). IEEE, pp 179–183. https://doi.org/10.1109/ISMAR-Adjunct.2018.00062. https://ieeexplore.ieee.org/document/8699172/

  8. Huang W, Alem L, Tecchia F (2013) SIGGRAPH Asia 2013 emerging technologies on—SA ’13. ACM Press, New York, pp 1–3. https://doi.org/10.1145/2542284.2542294. http://dl.acm.org/citation.cfm?doid=2542284.2542294

  9. Teo T, Lee GA, Billinghurst M, Adcock M (2018) Proceedings of the 30th Australian conference on computer–human interaction—OzCHI ’18. ACM Press, New York, pp 406–410. https://doi.org/10.1145/3292147.3292200. http://dl.acm.org/citation.cfm?doid=3292147.3292200

  10. Adcock M, Ranatunga D, Smith R, Thomas BH (2014) Proceedings of the 2nd ACM symposium on spatial user interaction—SUI ’14. ACM Press, New York, pp 113–122. https://doi.org/10.1145/2659766.2659768. http://dl.acm.org/citation.cfm?doid=2659766.2659768

  11. Lee GA, Teo T, Kim S, Billinghurst M (2017) Sharedsphere: MR collaboration through shared live panorama. https://doi.org/10.1145/3132818.3132827

  12. Adcock M, Anderson S, Thomas B (2013) RemoteFusion: real time depth camera fusion for remote collaboration on physical tasks. https://doi.org/10.1145/2534329.2534331. http://dl.acm.org/citation.cfm?doid=2534329.2534331

  13. Gauglitz S, Nuernberger B, Turk M, Höllerer T (2014) Proceedings of the 20th ACM symposium on virtual reality software and technology—VRST ’14, pp 197–205. https://doi.org/10.1145/2671015.2671016. http://dl.acm.org/citation.cfm?doid=2671015.2671016

  14. Teo T, Lawrence L, Lee GA, Billinghurst M, Adcock M (2019) Proceedings of the 2019 CHI conference on human factors in computing systems—CHI ’19. ACM Press, New York, pp 1–14. https://doi.org/10.1145/3290605.3300431. http://dl.acm.org/citation.cfm?doid=3290605.3300431

  15. Teo T, Hayati AF, Lee GA, Billinghurst M, Adcock M (2019) 25th ACM symposium on virtual reality software and technology on-VRST ’19. ACM Press, New York, pp 1–11. https://doi.org/10.1145/3359996.3364238. http://dl.acm.org/citation.cfm?doid=3359996.3364238

  16. Piumsomboon T, Day A, Ens B, Lee Y, Lee G, Billinghurst M (2017) Exploring enhancements for remote mixed reality collaboration. https://doi.org/10.1145/3132787.3139200

  17. Kasahara S, Rekimoto J (2015) Proceedings of the 21st ACM symposium on virtual reality software and technology, vol 23, no 3, p 217. https://doi.org/10.1145/2821592.2821608

  18. NP. Jouppi, NP (2002) Proceedings of the 2002 ACM conference on computer supported cooperative work—CSCW ’02. ACM Press, New York, p. 354. https://doi.org/10.1145/587078.587128. http://portal.acm.org/citation.cfm?doid=587078.587128

  19. Yamazawa K, Onoe Y, Yokoya N, Takemura H (2000) Systems and computers in Japan, vol 31, no 6, p 56. https://doi.org/10.1002/(SICI)1520-684X(200006)31:6<56::AID-SCJ6>3.0.CO;2-0. http://doi.wiley.com/10.1002/%28SICI%291520-684X%28200006%2931%3A6%3C56%3A%3AAID-SCJ6%3E3.0.CO%3B2-0

  20. Schulz R, Ward B, Roberts J (2013) ICRA 2013 workshop: human robot interaction (HRI) for assistance and industrial robots. Scientific knowledge, standards and regulatory framework. How do I design for the real world?, Karlsruhe, Germany, pp 37–38

  21. Matsuda A, Miyaki T, Rekimoto J (2017) Proceedings of the 8th augmented human international conference on-AH ’17. ACM Press, New York, pp 1–9. https://doi.org/10.1145/3041164.3041182. http://dl.acm.org/citation.cfm?doid=3041164.3041182

  22. Tang A, Fakourfar O, Neustaedter C, Bateman S (2017) Proceedings of the 2017 conference on designing interactive systems, pp 1327–1339. https://doi.org/10.1145/3064663.3064707

  23. Piumsomboon T, Lee GA, Irlitti A, Ens B, Thomas BH, Billinghurst M (2019) Proceedings of the 2019 CHI conference on human factors in computing systems—CHI ’19. ACM Press, New York, pp 1–17. https://doi.org/10.1145/3290605.3300458. http://dl.acm.org/citation.cfm?doid=3290605.3300458

  24. Tait M, Billinghurst M (2015) The effect of view independence in a collaborative AR system. Comput Support Coop Work (CSCW) 24(6):563. https://doi.org/10.1007/s10606-015-9231-8

    Article  Google Scholar 

  25. Lee GA, Teo T, Kim S, Billinghurst M (2018) 2018 IEEE international symposium on mixed and augmented reality (ISMAR). IEEE, pp 153–164. https://doi.org/10.1109/ISMAR.2018.00051. https://ieeexplore.ieee.org/document/8613761/

  26. Stotko P, Krumpen S, Hullin MB, Weinmann M, Klein R (2019) Slamcast: large-scale, real-time 3d reconstruction and streaming for immersive multi-client live telepresence. IEEE Trans Vis Comput Graph 25(5):2102. https://doi.org/10.1109/TVCG.2019.2899231

    Article  Google Scholar 

  27. Dai A, Nienerbner M, Zollhofer M, Izadi S, Theobalt C (2017) Bundlefusion: real-time globally consistent 3d reconstruction using on-the-fly surface reintegration. Trans ACM Graph 36(4):1. https://doi.org/10.1145/3072959.3054739

    Article  Google Scholar 

  28. Gauglitz S, Nuernberger B, Turk M, Höllerer T (2014) Proceedings of the 27th annual ACM symposium on User interface software and technology—UIST ’14. ACM Press, New York, pp 449–459. https://doi.org/10.1145/2642918.2647372. http://dl.acm.org/citation.cfm?doid=2642918.2647372

  29. Adcock M, Gunn C (2010) ACM SIGGRAPH ASIA. Posters on-SA ’10. ACM Press, New York, p 1. https://doi.org/10.1145/1900354.1900423. http://portal.acm.org/citation.cfm?doid=1900354.1900423

  30. Agisoft. PhotoScan. http://www.agisoft.com/

  31. HoloLens M https://www.microsoft.com/en-us/hololen

  32. Segal M, Korobkin C, van Widenfelt R, Foran J, Haeberli P (1992) Association for Computing Machinery (ACM), pp 249–252. https://doi.org/10.1145/133994.134071

  33. Kennedy RS, Lane KSBNE, Lilienthal Michael G (1993) Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. Int J Aviat Psychol 3(3):203

    Article  Google Scholar 

  34. Harms C, Biocca F (2004) Seventh annual international workshop: presence, p 246

  35. Vorderer P, Gouveia FR, Biocca F, Saari T, Jäncke F, Böcking S, Schramm H, Gysbers A, Hartmann T, Klimmt C, Laarni J, Ravaja N, Sacau A, Baumgartner T, Jäncke P, Wirth W (2004) MEC spatial presence questionnaire. MEC- SPQ), short documentation and instructions for application

  36. Sauro J, Dumas JS (2009) Comparison of three one-question, post-task usability questionnaires

  37. Zijlstra FRH (1993) Efficiency in work behaviour: a design approach for modern tools

  38. Brooke J (1996) SUS: a quick and dirty usability scale

  39. Wobbrock JO, Findlater L, Gergle D, Higgins JJ (2011) Proceedings of the 2011 annual conference on Human factors in computing systems—CHI ’11. ACM Press, New York, p 143. https://doi.org/10.1145/1978942.1978963. http://dl.acm.org/citation.cfm?doid=1978942.1978963

  40. Gauglitz S, Lee C, Turk M, Höllerer T (2012) Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services—MobileHCI ’12. ACM Press, New York, p 241. https://doi.org/10.1145/2371574.2371610. http://dl.acm.org/citation.cfm?doid=2371574.2371610

  41. Günther S, Kratz S, Avrahami D, Mühlhäuser M (2018) Proceedings of the 11th PErvasive technologies related to assistive environments conference on-PETRA ’18. ACM Press, New York, pp 339–344. https://doi.org/10.1145/3197768.3201568. http://dl.acm.org/citation.cfm?doid=3197768.3201568

  42. Rahimi Moghadam K, Banigan C, Ragan ED (2018) Scene transitions and teleportation in virtual reality and the implications for spatial awareness and sickness. IEEE Trans Vis Comput Graph. https://doi.org/10.1109/TVCG.2018.2884468

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Theophilus Teo.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Teo, T., Norman, M., Lee, G.A. et al. Exploring interaction techniques for 360 panoramas inside a 3D reconstructed scene for mixed reality remote collaboration. J Multimodal User Interfaces 14, 373–385 (2020). https://doi.org/10.1007/s12193-020-00343-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-020-00343-x

Keywords

Navigation