Abstract
Teleimmersion is an emerging technology that enables users to collaborate remotely by generating realistic 3D avatars in real time and rendering them inside a shared virtual space. The teleimmersive environment thus provides a venue for collaborative work on 3D data such as medical imaging, scientific data and models, archaeological datasets, architectural or mechanical designs, remote training (e.g., oil rigs, military applications), and remote teaching of physical activities (e.g., rehabilitation, dance). In this paper, we present our research work performed over the course of several years in developing the teleimmersive technology using image-based stereo and more recently Kinect. We outline the issues pertaining to the capture, transmission, rendering, and interaction. We describe several applications where we have explored the use of the 3D teleimmersion for remote interaction and collaboration among professional and scientific users. We believe the presented findings are relevant for future developers in teleimmersion and apply across various 3D video capturing technologies.
Similar content being viewed by others
References
Bailenson JN, Patel K, Nielsen A, Bajcsy R, Jung S, Kurillo G (2008) The effect of interactivity on learning physical actions in virtual reality. Media Psychol 11:354–376
Bajcsy P, McHenry K, Na HJ, Malik R, Spencer A, Lee SK, Kooper R, Frogley M (2009) Immersive environments for rehabilitation activities. In: Proceedings of the 17th ACM international conference on multimedia, ACM, New York, MM ’09, pp 829–832
Baker H, Tanguay D, Sobel I, Gelb D, Gross M, Culbertson W, Malzenbender T (2002) The coliseum immersive teleconferencing system. In: Proceedings of international workshop on immersive telepresence, Juan-les-Pins, France
Benford S, Greenhalgh C, Bowers J, Snowdon D, Fahlen LE (1995) User embodiment in collaborative virtual environments. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ’95), ACM Press/Addison-Wesley Publishing Co. New York, pp 242–249
Benko H, Jota R, Wilson A (2012) Miragetable: freehand interaction on a projected augmented reality tabletop. In: CHI, pp 199–208
CadFaster (2012) Cadfaster. http://www.cadfaster.com/index.php/Products
Cheng X, Davis J, Slusallek P (2000) Wide area camera calibration using virtual calibration objects. In: Proceedings of IEEE conference on computer vision and pattern recognition (CVPR 2000)
DeFanti T, Sandin D, Brown M, Pape D, Anstey J, Bogucki M, Dawe G, Johnson A, Huang TS (1999) Technologies for virtual reality/tele-immersion applications: issues of research in image display and global networking. In: EC/NSF workshop on research frontiers in virtual environments and human-centered computing
Delaney D, Ward T, McLoone S (2006) On consistency and network latency in distributed interactive applications: a survey–part I. Presence Teleoper Virtual Environ 15:218–234
De Silva DVSX, Fernando WAC, Kodikaraarachchi H, Worrall ST, Kondoz AM (2010) A depth map post-processing technique for 3D-TV systems based on compression artifact analysis. IEEE J Sel Top Signal Process November 2011
Doherty-Sneddon G, Anderson A, O’Malley C, Langton S, Garrod S, Bruce V (1997) Face-to-face and video-mediated communication: a comparison of dialogue structure and task performance. J Exp Psychol Appl 3(2):105–125
Eon (2009) Eon Reality: EON Coliseum. http://www.eonreality.com/product_coliseum.html
Forte M, Kurillo G (2010) Cyberarchaeology—experimenting with teleimmersive archaeology. In: Proceedings of 16th international conference on virtual systems and multimedia (VSMM 2010), Seoul
Forte M, Kurillo G, Matlock T (2010) Teleimmersive archaeology: simulation and cognitive impact. In: Ioannides M, Fellner AG D, Hadjimitsis D (eds) EuroMed 2010—digital heritage, Lemesos
Fry R, Smith G (1975) The effects of feedback and eye contact on performance of a digit-coding task. J Soc Psychol 96:145–146
Gross M, Würmlin S, Naef M, Lamboray E, Spagno C, Kunz A, Koller-Meier E, Svoboda T, Gool LV, Lang S, Strehlke K, Moere AV, Staadt O (2003) blue-c: a spatially immersive display and 3D video portal for telepresence. ACM Trans Graph 22(3):819–827
Gutwin C (2001) The effects of network delays on group work in real-time groupware. In: Proceedings of the seventh conference on European conference on computer supported cooperative work, Kluwer, Norwell, pp 299–318
Hasenfratz J, Lapierre M, Sillion F (2004) A real-time system for full-body interaction with virtual worlds. In: Proceedings of Eurographics symposium on virtual environments, The Eurographics Association, pp 147–156
Hypercosm (2009) Hypercosm. http://hypercosm.com
Ihrke I, Ahrenberg L, M Magnor M (2004) External camera calibration for synchronized multi-video systems. In: Proceedings of 12th international conference on computer graphics, visualization and computer vision 2004, vol 12, Plzen, pp 537–544
Jung S, Bajcsy R (2006) A framework for constructing real-time immersive environments for training physical activities. J Multime'd 1(7):9–17
Kalra P, Magnenat-Thalman N, Moccozet L, Sannier G, Aubel A, Thalman D (1998) Real-time animation of realistic virtual humans. IEEE Comput Graphics Appl 18(25):42–56
Kauff P, Schreer O (2002) An immersive 3D video-conferencing system using shared virtual team user environments. In: Proceedings of the 4th international conference on collaborative virtual environments, pp 105–112
Khoshelham K (2011) Accuracy analysis of Kinect depth data. In: Proceedings of ISPRS workshop laser scanning, Calgary
Knoblauch D, Font P, Kuester F (2010) VirtualizeMe: real-time avatar creation for tele-Immersion environments. In: Virtual reality conference (VR), 2010 IEEE, pp 279–280
Kolb A, Barth E, Koch R, Larsen R (2009) Time-of-flight sensors in computer graphics. In: Proceedings of Eurographics 2009—state of the art reports, pp 119–134
Kreylos O (2008) Environment-independent VR development. In: Bebis G, et al. (eds) Advances in visual computing, lecture notes in computer science. Springer, Berlin, pp 901–912
Kurillo G, Li Z, Bajcsy R (2008a) Wide-area external multi-camera calibration using vision graphs and virtual calibration object. In: Proceedings of 2nd ACM/IEEE international conference on distributed smart cameras (ICDSC 08), IEEE, Stanford
Kurillo G, Vasudevan R, Lobaton E, Bajcsy R (2008b) A framework for collaborative real-time 3D teleimmersion in a geographically distributed environment. In: Proceedings of 10th IEEE international symposium on multimedia (ISM 2008), Berkeley, pp 111–118
Kurillo G, Bajcsy R, Kreylos O, Rodriguez R (2009a) Teleimmersive environment for remote medical collaboration. In: Westwood J, et al. (eds) Medicine meets virtual reality 17, IOS press, Ohmsha, pp 148–150
Kurillo G, Li Z, Bajcsy R (2009b) Framework for hierarchical calibration of multi-camera systems for teleimmersion. In: Proceedings of IMMERSCOM 2009, Berkeley, CA, pp 1:1–1:6
Kurillo G, Forte M, Bajcsy R (2010a) Teleimmersive 3D collaborative environment for cyberarchaeology. In: IEEE/CVPR workshop, applications of computer vision in archaeology (ACVA 2010), San Francisco
Kurillo G, Koritnik T, Bajd T, Bajcsy R (2010b) Real-time 3D avatars for tele-rehabilitation in virtual reality. In: Westwood J, Westwood S, et al. (eds) Proceedings of 18th medicine meets virtual reality (MMVR) conference., IOS press, pp 290–296
Kuster C, Popa T, Zach C, Gotsman C, Gross M (2011) Freecam: A hybrid camera system for interactive free-viewpoint video. In: Proceedings of vision, modeling, and visualization (VMV)
Ladikos A, Benhimane S, Navab N (2008) Efficient visual hull computation for real-time 3D reconstruction using CUDA. In: IEEE computer society conference on computer vision and pattern recognition workshops, 2008. CVPR Workshops, pp 1–8
libjpeg (2012) libjpeg-turbo. http://libjpeg-turbo.virtualgl.org
Litos G, Zabulis X, Triantafyllidis G (2006) Synchronous image acquisition based on network synchronization. In: Computer vision and pattern recognition workshop 2006. CVPRW ’06. Conference on, pp 167–173
Maimone A, Fuchs H (2011) Encumbrance-free telepresence system with real-time 3D capture and display using commodity depth cameras. In: Proceedings of the 2011 10th IEEE international symposium on mixed and augmented reality, IEEE Computer Society, Washington, ISMAR ’11, pp 137–146
Maimone A, Fuchs H (2012) Reducing interference between multiple structured light depth sensors using motion. In: Virtual reality workshops (VR), IEEE, pp 51–54
Microsoft (2010) Microsoft Kinect. http://www.xbox.com/en-US/kinect
Mulligan J, Daniilidis K (2001) Real time trinocular stereo for tele-immersion. In: Proceedings of 2001 international conference on image processing, Thessaloniki, pp 959–962
Nahrstedt K, Bajcsy R, Wymore L, Kurillo G, Mezur K, Sheppard R, Yang Z, Wu W (2007) Symbiosis of tele-immersive environments with creative choreography. In: ACM workshop on supporting creative acts beyond dissemination, associated with 6th ACM creativity and cognition conference, Washington
Nintendo (2006) Nintendo Wii. http://www.nintendo.com/wii
Obdrzalek S, Kurilo G, Han J, Abresch T, Bajcsy R (2012) Real-time human pose detection and tracking for tele-rehabilitation in virtual reality. In: Proceedings of the 19th medicine meets virtual reality conference (MMVR), Newport Beach
OpenWonderland (2010) Sun Systems, openWonderLand. http://openwonderland.org
Petit B, Lesage JD, Menier C, Allard J, Franco JS, Raffin B, Boyer E, Faure F (2010) Multicamera real-time 3D modeling for telepresence and remote collaboration. Int J Digit Multime'd Broadcast 12
Schröder Y, Scholz A, Berger K, Ruhl K, Guthe S, Magnor M (2011) Multiple kinect studies. Tech Rep 09–15, ICG
SecondLife (2003) SecondLife. http://secondlife.com
Sheppard R, Wu W, Yang Z, Nahrstedt K, Wymore L, Kurillo G, Bajcsy R, Mezur K (2007) New digital options in geographically distributed dance collaborations with TEEVE: tele-immersive environments for everybody. In: Proceedings of the 15th international conference on multimedia, ACM, New York, MULTIMEDIA ’07, pp 1085–1086
Svoboda T, Martinec D, Pajdla T (2005) A convenient multicamera self-calibration for virtual environments. Presence 14(4):407–422
Vasudevan R, Lobaton E, Kurillo G, Bajcsy R, Bernardin T, Hamann B, Nahrstedt K (2010a) A methodology for remote virtual interaction in teleimmersive environments. In: Proceedings of first ACM multimedia systems conference, Scottsdale, pp 281–292
Vasudevan R, Zhou Z, Kurillo G, Lobaton E, Bajcsy R, Nahrstedt K (2010b) Real-time stereo-vision system for 3D teleimmersive collaboration. In: Proceedings of IEEE international conference on multimedia & expo (ICME 2010), Singapore, pp 1208–1213
Vasudevan R, Kurillo G, Lobaton E, Bernardin T, Kreylos O, Bajcsy R, Nahrstedt K (2011) High quality visualization for geographically distributed 3D teleimmersive applications. IEEE Trans Multime'd 13(3):573–584
Waschbüsch M, Würmlin S, Cotting D, Sadlo F, Gross MH (2005) Scalable 3D video of dynamic scenes. Vis Comput 21(8-10):629–638
Wu W, Arefin MA, Huang Z, Agarwal P, Shi S, Rivas R, Nahrstedt K (2010) “I’m the Jedi!”—a case study of user experience in 3D tele-immersive gaming. In: ISM, pp 220–227
Wu W, Arefin A, Kurillo G, Agarwal P, Nahrstedt K, Bajcsy R (2011) Color-plus-depth level-of-details in 3D teleimmersive video—a psychophysical approach. In: Proceedings of ACM multimedia
Yang Y, Wang X, Chen JX (2002) Rendering avatars in virtual reality: integrating a 3D model with 2D images. Comput Sci Eng 4(1):86–91
Wu W, Arefin A, Rivas R, Nahrstedt K, Sheppard R, Yang Z (2009) Quality of experience in distributed interactive multimedia environments: toward a theoretical framework. In: Proceedings of ACM multimedia, Beijing, pp 481-490
Yang Z, Nahrstedt K, Cui Y, Yu B, Liang J, Jung SH, Bajcsy R (2005) TEEVE: The next generation architecture for tele-immersive environment. In: Seventh IEEE international symposium on multimedia (ISM 2005). IEEE Computer Society, Irvine, pp 112–119
Yang Z, Wu W, Nahrstedt K, Kurillo G, Bajcsy R (2010) Enabling multi-party 3D tele-immersive environments with viewcast. ACM transactions on multimedia computing, communications, and applications (TOMCCAP) 6:1–30
Yang Z, Wu W, Nahrstedt K, Kurillo G, Bajscy R (2007) ViewCast: view dissemination and management for multi-party 3D tele-immersive environments. In: Proceedings of ACM multimedia, Augsburg, pp 882–891
Zhang D, Nomura Y, Fujii S (1991) Error analysis and optimization of camera calibration. In: Proceedings of IEEE/RSJ international workshop on intelligent robots and systems (IROS 91), Osaka, pp 292–296
Acknowledgments
We wish to thank Ram Vasudevan and Edgar Lobaton, University of California at Berkeley, for contribution on the stereo reconstruction and Zhong Zhou, University of Beijing, for texture compression. We also thank Tony Bernardin and Oliver Kreylos, University of California at Davis, for the implementation of the rendering and support in integration with the Vrui framework. Furthermore, we thank all of our past and current collaborators, including Jeremy Bailenson (Stanford University), Maurizio Forte (UCM), Jay Han (UCDMC), Louise Kellogg (UCD), Klara Nahrstedt (UIUC), and Lisa Wymore (UCB). This work was supported in part by the National Science Foundation (NSF grants: #0703787, #0724681, #0840399, #1111965), HP Labs, the European Aeronautic Defence and Space Company (EADS), and the Center for Information Technology in the Interest of Society (CITRIS) at University of California, Berkeley. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Kurillo, G., Bajcsy, R. 3D teleimmersion for collaboration and interaction of geographically distributed users. Virtual Reality 17, 29–43 (2013). https://doi.org/10.1007/s10055-012-0217-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10055-012-0217-2