Skip to main content

3D teleimmersion for collaboration and interaction of geographically distributed users

Virtual Reality volume 17, pages 29–43 (2013)Cite this article

Abstract

Teleimmersion is an emerging technology that enables users to collaborate remotely by generating realistic 3D avatars in real time and rendering them inside a shared virtual space. The teleimmersive environment thus provides a venue for collaborative work on 3D data such as medical imaging, scientific data and models, archaeological datasets, architectural or mechanical designs, remote training (e.g., oil rigs, military applications), and remote teaching of physical activities (e.g., rehabilitation, dance). In this paper, we present our research work performed over the course of several years in developing the teleimmersive technology using image-based stereo and more recently Kinect. We outline the issues pertaining to the capture, transmission, rendering, and interaction. We describe several applications where we have explored the use of the 3D teleimmersion for remote interaction and collaboration among professional and scientific users. We believe the presented findings are relevant for future developers in teleimmersion and apply across various 3D video capturing technologies.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

References

  • Bailenson JN, Patel K, Nielsen A, Bajcsy R, Jung S, Kurillo G (2008) The effect of interactivity on learning physical actions in virtual reality. Media Psychol 11:354–376

    Article  Google Scholar 

  • Bajcsy P, McHenry K, Na HJ, Malik R, Spencer A, Lee SK, Kooper R, Frogley M (2009) Immersive environments for rehabilitation activities. In: Proceedings of the 17th ACM international conference on multimedia, ACM, New York, MM ’09, pp 829–832

  • Baker H, Tanguay D, Sobel I, Gelb D, Gross M, Culbertson W, Malzenbender T (2002) The coliseum immersive teleconferencing system. In: Proceedings of international workshop on immersive telepresence, Juan-les-Pins, France

  • Benford S, Greenhalgh C, Bowers J, Snowdon D, Fahlen LE (1995) User embodiment in collaborative virtual environments. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ’95), ACM Press/Addison-Wesley Publishing Co. New York, pp 242–249

  • Benko H, Jota R, Wilson A (2012) Miragetable: freehand interaction on a projected augmented reality tabletop. In: CHI, pp 199–208

  • CadFaster (2012) Cadfaster. http://www.cadfaster.com/index.php/Products

  • Cheng X, Davis J, Slusallek P (2000) Wide area camera calibration using virtual calibration objects. In: Proceedings of IEEE conference on computer vision and pattern recognition (CVPR 2000)

  • DeFanti T, Sandin D, Brown M, Pape D, Anstey J, Bogucki M, Dawe G, Johnson A, Huang TS (1999) Technologies for virtual reality/tele-immersion applications: issues of research in image display and global networking. In: EC/NSF workshop on research frontiers in virtual environments and human-centered computing

  • Delaney D, Ward T, McLoone S (2006) On consistency and network latency in distributed interactive applications: a survey–part I. Presence Teleoper Virtual Environ 15:218–234

    Article  Google Scholar 

  • De Silva DVSX, Fernando WAC, Kodikaraarachchi H, Worrall ST, Kondoz AM (2010) A depth map post-processing technique for 3D-TV systems based on compression artifact analysis. IEEE J Sel Top Signal Process November 2011

  • Doherty-Sneddon G, Anderson A, O’Malley C, Langton S, Garrod S, Bruce V (1997) Face-to-face and video-mediated communication: a comparison of dialogue structure and task performance. J Exp Psychol Appl 3(2):105–125

    Article  Google Scholar 

  • Eon (2009) Eon Reality: EON Coliseum. http://www.eonreality.com/product_coliseum.html

  • Forte M, Kurillo G (2010) Cyberarchaeology—experimenting with teleimmersive archaeology. In: Proceedings of 16th international conference on virtual systems and multimedia (VSMM 2010), Seoul

  • Forte M, Kurillo G, Matlock T (2010) Teleimmersive archaeology: simulation and cognitive impact. In: Ioannides M, Fellner AG D, Hadjimitsis D (eds) EuroMed 2010—digital heritage, Lemesos

  • Fry R, Smith G (1975) The effects of feedback and eye contact on performance of a digit-coding task. J Soc Psychol 96:145–146

    Article  Google Scholar 

  • Gross M, Würmlin S, Naef M, Lamboray E, Spagno C, Kunz A, Koller-Meier E, Svoboda T, Gool LV, Lang S, Strehlke K, Moere AV, Staadt O (2003) blue-c: a spatially immersive display and 3D video portal for telepresence. ACM Trans Graph 22(3):819–827

    Article  Google Scholar 

  • Gutwin C (2001) The effects of network delays on group work in real-time groupware. In: Proceedings of the seventh conference on European conference on computer supported cooperative work, Kluwer, Norwell, pp 299–318

  • Hasenfratz J, Lapierre M, Sillion F (2004) A real-time system for full-body interaction with virtual worlds. In: Proceedings of Eurographics symposium on virtual environments, The Eurographics Association, pp 147–156

  • Hypercosm (2009) Hypercosm. http://hypercosm.com

  • Ihrke I, Ahrenberg L, M Magnor M (2004) External camera calibration for synchronized multi-video systems. In: Proceedings of 12th international conference on computer graphics, visualization and computer vision 2004, vol 12, Plzen, pp 537–544

  • Jung S, Bajcsy R (2006) A framework for constructing real-time immersive environments for training physical activities. J Multime'd 1(7):9–17

    Google Scholar 

  • Kalra P, Magnenat-Thalman N, Moccozet L, Sannier G, Aubel A, Thalman D (1998) Real-time animation of realistic virtual humans. IEEE Comput Graphics Appl 18(25):42–56

    Article  Google Scholar 

  • Kauff P, Schreer O (2002) An immersive 3D video-conferencing system using shared virtual team user environments. In: Proceedings of the 4th international conference on collaborative virtual environments, pp 105–112

  • Khoshelham K (2011) Accuracy analysis of Kinect depth data. In: Proceedings of ISPRS workshop laser scanning, Calgary

  • Knoblauch D, Font P, Kuester F (2010) VirtualizeMe: real-time avatar creation for tele-Immersion environments. In: Virtual reality conference (VR), 2010 IEEE, pp 279–280

  • Kolb A, Barth E, Koch R, Larsen R (2009) Time-of-flight sensors in computer graphics. In: Proceedings of Eurographics 2009—state of the art reports, pp 119–134

  • Kreylos O (2008) Environment-independent VR development. In: Bebis G, et al. (eds) Advances in visual computing, lecture notes in computer science. Springer, Berlin, pp 901–912

  • Kurillo G, Li Z, Bajcsy R (2008a) Wide-area external multi-camera calibration using vision graphs and virtual calibration object. In: Proceedings of 2nd ACM/IEEE international conference on distributed smart cameras (ICDSC 08), IEEE, Stanford

  • Kurillo G, Vasudevan R, Lobaton E, Bajcsy R (2008b) A framework for collaborative real-time 3D teleimmersion in a geographically distributed environment. In: Proceedings of 10th IEEE international symposium on multimedia (ISM 2008), Berkeley, pp 111–118

  • Kurillo G, Bajcsy R, Kreylos O, Rodriguez R (2009a) Teleimmersive environment for remote medical collaboration. In: Westwood J, et al. (eds) Medicine meets virtual reality 17, IOS press, Ohmsha, pp 148–150

  • Kurillo G, Li Z, Bajcsy R (2009b) Framework for hierarchical calibration of multi-camera systems for teleimmersion. In: Proceedings of IMMERSCOM 2009, Berkeley, CA, pp 1:1–1:6

  • Kurillo G, Forte M, Bajcsy R (2010a) Teleimmersive 3D collaborative environment for cyberarchaeology. In: IEEE/CVPR workshop, applications of computer vision in archaeology (ACVA 2010), San Francisco

  • Kurillo G, Koritnik T, Bajd T, Bajcsy R (2010b) Real-time 3D avatars for tele-rehabilitation in virtual reality. In: Westwood J, Westwood S, et al. (eds) Proceedings of 18th medicine meets virtual reality (MMVR) conference., IOS press, pp 290–296

  • Kuster C, Popa T, Zach C, Gotsman C, Gross M (2011) Freecam: A hybrid camera system for interactive free-viewpoint video. In: Proceedings of vision, modeling, and visualization (VMV)

  • Ladikos A, Benhimane S, Navab N (2008) Efficient visual hull computation for real-time 3D reconstruction using CUDA. In: IEEE computer society conference on computer vision and pattern recognition workshops, 2008. CVPR Workshops, pp 1–8

  • libjpeg (2012) libjpeg-turbo. http://libjpeg-turbo.virtualgl.org

  • Litos G, Zabulis X, Triantafyllidis G (2006) Synchronous image acquisition based on network synchronization. In: Computer vision and pattern recognition workshop 2006. CVPRW ’06. Conference on, pp 167–173

  • Maimone A, Fuchs H (2011) Encumbrance-free telepresence system with real-time 3D capture and display using commodity depth cameras. In: Proceedings of the 2011 10th IEEE international symposium on mixed and augmented reality, IEEE Computer Society, Washington, ISMAR ’11, pp 137–146

  • Maimone A, Fuchs H (2012) Reducing interference between multiple structured light depth sensors using motion. In: Virtual reality workshops (VR), IEEE, pp 51–54

  • Microsoft (2010) Microsoft Kinect. http://www.xbox.com/en-US/kinect

  • Mulligan J, Daniilidis K (2001) Real time trinocular stereo for tele-immersion. In: Proceedings of 2001 international conference on image processing, Thessaloniki, pp 959–962

  • Nahrstedt K, Bajcsy R, Wymore L, Kurillo G, Mezur K, Sheppard R, Yang Z, Wu W (2007) Symbiosis of tele-immersive environments with creative choreography. In: ACM workshop on supporting creative acts beyond dissemination, associated with 6th ACM creativity and cognition conference, Washington

  • Nintendo (2006) Nintendo Wii. http://www.nintendo.com/wii

  • Obdrzalek S, Kurilo G, Han J, Abresch T, Bajcsy R (2012) Real-time human pose detection and tracking for tele-rehabilitation in virtual reality. In: Proceedings of the 19th medicine meets virtual reality conference (MMVR), Newport Beach

  • OpenWonderland (2010) Sun Systems, openWonderLand. http://openwonderland.org

  • Petit B, Lesage JD, Menier C, Allard J, Franco JS, Raffin B, Boyer E, Faure F (2010) Multicamera real-time 3D modeling for telepresence and remote collaboration. Int J Digit Multime'd Broadcast 12

  • Schröder Y, Scholz A, Berger K, Ruhl K, Guthe S, Magnor M (2011) Multiple kinect studies. Tech Rep 09–15, ICG

  • SecondLife (2003) SecondLife. http://secondlife.com

  • Sheppard R, Wu W, Yang Z, Nahrstedt K, Wymore L, Kurillo G, Bajcsy R, Mezur K (2007) New digital options in geographically distributed dance collaborations with TEEVE: tele-immersive environments for everybody. In: Proceedings of the 15th international conference on multimedia, ACM, New York, MULTIMEDIA ’07, pp 1085–1086

  • Svoboda T, Martinec D, Pajdla T (2005) A convenient multicamera self-calibration for virtual environments. Presence 14(4):407–422

    Article  Google Scholar 

  • Vasudevan R, Lobaton E, Kurillo G, Bajcsy R, Bernardin T, Hamann B, Nahrstedt K (2010a) A methodology for remote virtual interaction in teleimmersive environments. In: Proceedings of first ACM multimedia systems conference, Scottsdale, pp 281–292

  • Vasudevan R, Zhou Z, Kurillo G, Lobaton E, Bajcsy R, Nahrstedt K (2010b) Real-time stereo-vision system for 3D teleimmersive collaboration. In: Proceedings of IEEE international conference on multimedia & expo (ICME 2010), Singapore, pp 1208–1213

  • Vasudevan R, Kurillo G, Lobaton E, Bernardin T, Kreylos O, Bajcsy R, Nahrstedt K (2011) High quality visualization for geographically distributed 3D teleimmersive applications. IEEE Trans Multime'd 13(3):573–584

    Article  Google Scholar 

  • Waschbüsch M, Würmlin S, Cotting D, Sadlo F, Gross MH (2005) Scalable 3D video of dynamic scenes. Vis Comput 21(8-10):629–638

    Article  Google Scholar 

  • Wu W, Arefin MA, Huang Z, Agarwal P, Shi S, Rivas R, Nahrstedt K (2010) “I’m the Jedi!”—a case study of user experience in 3D tele-immersive gaming. In: ISM, pp 220–227

  • Wu W, Arefin A, Kurillo G, Agarwal P, Nahrstedt K, Bajcsy R (2011) Color-plus-depth level-of-details in 3D teleimmersive video—a psychophysical approach. In: Proceedings of ACM multimedia

  • Yang Y, Wang X, Chen JX (2002) Rendering avatars in virtual reality: integrating a 3D model with 2D images. Comput Sci Eng 4(1):86–91

    MathSciNet  Article  Google Scholar 

  • Wu W, Arefin A, Rivas R, Nahrstedt K, Sheppard R, Yang Z (2009) Quality of experience in distributed interactive multimedia environments: toward a theoretical framework. In: Proceedings of ACM multimedia, Beijing, pp 481-490

  • Yang Z, Nahrstedt K, Cui Y, Yu B, Liang J, Jung SH, Bajcsy R (2005) TEEVE: The next generation architecture for tele-immersive environment. In: Seventh IEEE international symposium on multimedia (ISM 2005). IEEE Computer Society, Irvine, pp 112–119

  • Yang Z, Wu W, Nahrstedt K, Kurillo G, Bajcsy R (2010) Enabling multi-party 3D tele-immersive environments with viewcast. ACM transactions on multimedia computing, communications, and applications (TOMCCAP) 6:1–30

    Google Scholar 

  • Yang Z, Wu W, Nahrstedt K, Kurillo G, Bajscy R (2007) ViewCast: view dissemination and management for multi-party 3D tele-immersive environments. In: Proceedings of ACM multimedia, Augsburg, pp 882–891

  • Zhang D, Nomura Y, Fujii S (1991) Error analysis and optimization of camera calibration. In: Proceedings of IEEE/RSJ international workshop on intelligent robots and systems (IROS 91), Osaka, pp 292–296

Download references

Acknowledgments

We wish to thank Ram Vasudevan and Edgar Lobaton, University of California at Berkeley, for contribution on the stereo reconstruction and Zhong Zhou, University of Beijing, for texture compression. We also thank Tony Bernardin and Oliver Kreylos, University of California at Davis, for the implementation of the rendering and support in integration with the Vrui framework. Furthermore, we thank all of our past and current collaborators, including Jeremy Bailenson (Stanford University), Maurizio Forte (UCM), Jay Han (UCDMC), Louise Kellogg (UCD), Klara Nahrstedt (UIUC), and Lisa Wymore (UCB). This work was supported in part by the National Science Foundation (NSF grants: #0703787, #0724681, #0840399, #1111965), HP Labs, the European Aeronautic Defence and Space Company (EADS), and the Center for Information Technology in the Interest of Society (CITRIS) at University of California, Berkeley. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Author information

Authors and Affiliations

  1. Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, Berkeley, CA, 94720-1764, USA

    Gregorij Kurillo & Ruzena Bajcsy

Authors
  1. Gregorij Kurillo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ruzena Bajcsy
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gregorij Kurillo.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Kurillo, G., Bajcsy, R. 3D teleimmersion for collaboration and interaction of geographically distributed users. Virtual Reality 17, 29–43 (2013). https://doi.org/10.1007/s10055-012-0217-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10055-012-0217-2

Keywords

  • 3D video
  • 3D teleimmersion
  • Human–computer interaction
  • Remote collaboration
  • Telepresence