Virtual Reality

, Volume 17, Issue 1, pp 29–43 | Cite as

3D teleimmersion for collaboration and interaction of geographically distributed users

Original Article

Abstract

Teleimmersion is an emerging technology that enables users to collaborate remotely by generating realistic 3D avatars in real time and rendering them inside a shared virtual space. The teleimmersive environment thus provides a venue for collaborative work on 3D data such as medical imaging, scientific data and models, archaeological datasets, architectural or mechanical designs, remote training (e.g., oil rigs, military applications), and remote teaching of physical activities (e.g., rehabilitation, dance). In this paper, we present our research work performed over the course of several years in developing the teleimmersive technology using image-based stereo and more recently Kinect. We outline the issues pertaining to the capture, transmission, rendering, and interaction. We describe several applications where we have explored the use of the 3D teleimmersion for remote interaction and collaboration among professional and scientific users. We believe the presented findings are relevant for future developers in teleimmersion and apply across various 3D video capturing technologies.

Keywords

3D video 3D teleimmersion Human–computer interaction Remote collaboration Telepresence 

References

  1. Bailenson JN, Patel K, Nielsen A, Bajcsy R, Jung S, Kurillo G (2008) The effect of interactivity on learning physical actions in virtual reality. Media Psychol 11:354–376CrossRefGoogle Scholar
  2. Bajcsy P, McHenry K, Na HJ, Malik R, Spencer A, Lee SK, Kooper R, Frogley M (2009) Immersive environments for rehabilitation activities. In: Proceedings of the 17th ACM international conference on multimedia, ACM, New York, MM ’09, pp 829–832Google Scholar
  3. Baker H, Tanguay D, Sobel I, Gelb D, Gross M, Culbertson W, Malzenbender T (2002) The coliseum immersive teleconferencing system. In: Proceedings of international workshop on immersive telepresence, Juan-les-Pins, FranceGoogle Scholar
  4. Benford S, Greenhalgh C, Bowers J, Snowdon D, Fahlen LE (1995) User embodiment in collaborative virtual environments. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ’95), ACM Press/Addison-Wesley Publishing Co. New York, pp 242–249Google Scholar
  5. Benko H, Jota R, Wilson A (2012) Miragetable: freehand interaction on a projected augmented reality tabletop. In: CHI, pp 199–208Google Scholar
  6. CadFaster (2012) Cadfaster. http://www.cadfaster.com/index.php/Products
  7. Cheng X, Davis J, Slusallek P (2000) Wide area camera calibration using virtual calibration objects. In: Proceedings of IEEE conference on computer vision and pattern recognition (CVPR 2000)Google Scholar
  8. DeFanti T, Sandin D, Brown M, Pape D, Anstey J, Bogucki M, Dawe G, Johnson A, Huang TS (1999) Technologies for virtual reality/tele-immersion applications: issues of research in image display and global networking. In: EC/NSF workshop on research frontiers in virtual environments and human-centered computingGoogle Scholar
  9. Delaney D, Ward T, McLoone S (2006) On consistency and network latency in distributed interactive applications: a survey–part I. Presence Teleoper Virtual Environ 15:218–234CrossRefGoogle Scholar
  10. De Silva DVSX, Fernando WAC, Kodikaraarachchi H, Worrall ST, Kondoz AM (2010) A depth map post-processing technique for 3D-TV systems based on compression artifact analysis. IEEE J Sel Top Signal Process November 2011Google Scholar
  11. Doherty-Sneddon G, Anderson A, O’Malley C, Langton S, Garrod S, Bruce V (1997) Face-to-face and video-mediated communication: a comparison of dialogue structure and task performance. J Exp Psychol Appl 3(2):105–125CrossRefGoogle Scholar
  12. Eon (2009) Eon Reality: EON Coliseum. http://www.eonreality.com/product_coliseum.html
  13. Forte M, Kurillo G (2010) Cyberarchaeology—experimenting with teleimmersive archaeology. In: Proceedings of 16th international conference on virtual systems and multimedia (VSMM 2010), SeoulGoogle Scholar
  14. Forte M, Kurillo G, Matlock T (2010) Teleimmersive archaeology: simulation and cognitive impact. In: Ioannides M, Fellner AG D, Hadjimitsis D (eds) EuroMed 2010—digital heritage, LemesosGoogle Scholar
  15. Fry R, Smith G (1975) The effects of feedback and eye contact on performance of a digit-coding task. J Soc Psychol 96:145–146CrossRefGoogle Scholar
  16. Gross M, Würmlin S, Naef M, Lamboray E, Spagno C, Kunz A, Koller-Meier E, Svoboda T, Gool LV, Lang S, Strehlke K, Moere AV, Staadt O (2003) blue-c: a spatially immersive display and 3D video portal for telepresence. ACM Trans Graph 22(3):819–827CrossRefGoogle Scholar
  17. Gutwin C (2001) The effects of network delays on group work in real-time groupware. In: Proceedings of the seventh conference on European conference on computer supported cooperative work, Kluwer, Norwell, pp 299–318Google Scholar
  18. Hasenfratz J, Lapierre M, Sillion F (2004) A real-time system for full-body interaction with virtual worlds. In: Proceedings of Eurographics symposium on virtual environments, The Eurographics Association, pp 147–156Google Scholar
  19. Hypercosm (2009) Hypercosm. http://hypercosm.com
  20. Ihrke I, Ahrenberg L, M Magnor M (2004) External camera calibration for synchronized multi-video systems. In: Proceedings of 12th international conference on computer graphics, visualization and computer vision 2004, vol 12, Plzen, pp 537–544Google Scholar
  21. Jung S, Bajcsy R (2006) A framework for constructing real-time immersive environments for training physical activities. J Multime'd 1(7):9–17Google Scholar
  22. Kalra P, Magnenat-Thalman N, Moccozet L, Sannier G, Aubel A, Thalman D (1998) Real-time animation of realistic virtual humans. IEEE Comput Graphics Appl 18(25):42–56CrossRefGoogle Scholar
  23. Kauff P, Schreer O (2002) An immersive 3D video-conferencing system using shared virtual team user environments. In: Proceedings of the 4th international conference on collaborative virtual environments, pp 105–112Google Scholar
  24. Khoshelham K (2011) Accuracy analysis of Kinect depth data. In: Proceedings of ISPRS workshop laser scanning, CalgaryGoogle Scholar
  25. Knoblauch D, Font P, Kuester F (2010) VirtualizeMe: real-time avatar creation for tele-Immersion environments. In: Virtual reality conference (VR), 2010 IEEE, pp 279–280Google Scholar
  26. Kolb A, Barth E, Koch R, Larsen R (2009) Time-of-flight sensors in computer graphics. In: Proceedings of Eurographics 2009—state of the art reports, pp 119–134Google Scholar
  27. Kreylos O (2008) Environment-independent VR development. In: Bebis G, et al. (eds) Advances in visual computing, lecture notes in computer science. Springer, Berlin, pp 901–912Google Scholar
  28. Kurillo G, Li Z, Bajcsy R (2008a) Wide-area external multi-camera calibration using vision graphs and virtual calibration object. In: Proceedings of 2nd ACM/IEEE international conference on distributed smart cameras (ICDSC 08), IEEE, StanfordGoogle Scholar
  29. Kurillo G, Vasudevan R, Lobaton E, Bajcsy R (2008b) A framework for collaborative real-time 3D teleimmersion in a geographically distributed environment. In: Proceedings of 10th IEEE international symposium on multimedia (ISM 2008), Berkeley, pp 111–118Google Scholar
  30. Kurillo G, Bajcsy R, Kreylos O, Rodriguez R (2009a) Teleimmersive environment for remote medical collaboration. In: Westwood J, et al. (eds) Medicine meets virtual reality 17, IOS press, Ohmsha, pp 148–150Google Scholar
  31. Kurillo G, Li Z, Bajcsy R (2009b) Framework for hierarchical calibration of multi-camera systems for teleimmersion. In: Proceedings of IMMERSCOM 2009, Berkeley, CA, pp 1:1–1:6Google Scholar
  32. Kurillo G, Forte M, Bajcsy R (2010a) Teleimmersive 3D collaborative environment for cyberarchaeology. In: IEEE/CVPR workshop, applications of computer vision in archaeology (ACVA 2010), San FranciscoGoogle Scholar
  33. Kurillo G, Koritnik T, Bajd T, Bajcsy R (2010b) Real-time 3D avatars for tele-rehabilitation in virtual reality. In: Westwood J, Westwood S, et al. (eds) Proceedings of 18th medicine meets virtual reality (MMVR) conference., IOS press, pp 290–296Google Scholar
  34. Kuster C, Popa T, Zach C, Gotsman C, Gross M (2011) Freecam: A hybrid camera system for interactive free-viewpoint video. In: Proceedings of vision, modeling, and visualization (VMV)Google Scholar
  35. Ladikos A, Benhimane S, Navab N (2008) Efficient visual hull computation for real-time 3D reconstruction using CUDA. In: IEEE computer society conference on computer vision and pattern recognition workshops, 2008. CVPR Workshops, pp 1–8Google Scholar
  36. libjpeg (2012) libjpeg-turbo. http://libjpeg-turbo.virtualgl.org
  37. Litos G, Zabulis X, Triantafyllidis G (2006) Synchronous image acquisition based on network synchronization. In: Computer vision and pattern recognition workshop 2006. CVPRW ’06. Conference on, pp 167–173Google Scholar
  38. Maimone A, Fuchs H (2011) Encumbrance-free telepresence system with real-time 3D capture and display using commodity depth cameras. In: Proceedings of the 2011 10th IEEE international symposium on mixed and augmented reality, IEEE Computer Society, Washington, ISMAR ’11, pp 137–146Google Scholar
  39. Maimone A, Fuchs H (2012) Reducing interference between multiple structured light depth sensors using motion. In: Virtual reality workshops (VR), IEEE, pp 51–54Google Scholar
  40. Microsoft (2010) Microsoft Kinect. http://www.xbox.com/en-US/kinect
  41. Mulligan J, Daniilidis K (2001) Real time trinocular stereo for tele-immersion. In: Proceedings of 2001 international conference on image processing, Thessaloniki, pp 959–962Google Scholar
  42. Nahrstedt K, Bajcsy R, Wymore L, Kurillo G, Mezur K, Sheppard R, Yang Z, Wu W (2007) Symbiosis of tele-immersive environments with creative choreography. In: ACM workshop on supporting creative acts beyond dissemination, associated with 6th ACM creativity and cognition conference, WashingtonGoogle Scholar
  43. Nintendo (2006) Nintendo Wii. http://www.nintendo.com/wii
  44. Obdrzalek S, Kurilo G, Han J, Abresch T, Bajcsy R (2012) Real-time human pose detection and tracking for tele-rehabilitation in virtual reality. In: Proceedings of the 19th medicine meets virtual reality conference (MMVR), Newport BeachGoogle Scholar
  45. OpenWonderland (2010) Sun Systems, openWonderLand. http://openwonderland.org
  46. Petit B, Lesage JD, Menier C, Allard J, Franco JS, Raffin B, Boyer E, Faure F (2010) Multicamera real-time 3D modeling for telepresence and remote collaboration. Int J Digit Multime'd Broadcast 12Google Scholar
  47. Schröder Y, Scholz A, Berger K, Ruhl K, Guthe S, Magnor M (2011) Multiple kinect studies. Tech Rep 09–15, ICGGoogle Scholar
  48. SecondLife (2003) SecondLife. http://secondlife.com
  49. Sheppard R, Wu W, Yang Z, Nahrstedt K, Wymore L, Kurillo G, Bajcsy R, Mezur K (2007) New digital options in geographically distributed dance collaborations with TEEVE: tele-immersive environments for everybody. In: Proceedings of the 15th international conference on multimedia, ACM, New York, MULTIMEDIA ’07, pp 1085–1086Google Scholar
  50. Svoboda T, Martinec D, Pajdla T (2005) A convenient multicamera self-calibration for virtual environments. Presence 14(4):407–422CrossRefGoogle Scholar
  51. Vasudevan R, Lobaton E, Kurillo G, Bajcsy R, Bernardin T, Hamann B, Nahrstedt K (2010a) A methodology for remote virtual interaction in teleimmersive environments. In: Proceedings of first ACM multimedia systems conference, Scottsdale, pp 281–292Google Scholar
  52. Vasudevan R, Zhou Z, Kurillo G, Lobaton E, Bajcsy R, Nahrstedt K (2010b) Real-time stereo-vision system for 3D teleimmersive collaboration. In: Proceedings of IEEE international conference on multimedia & expo (ICME 2010), Singapore, pp 1208–1213Google Scholar
  53. Vasudevan R, Kurillo G, Lobaton E, Bernardin T, Kreylos O, Bajcsy R, Nahrstedt K (2011) High quality visualization for geographically distributed 3D teleimmersive applications. IEEE Trans Multime'd 13(3):573–584CrossRefGoogle Scholar
  54. Waschbüsch M, Würmlin S, Cotting D, Sadlo F, Gross MH (2005) Scalable 3D video of dynamic scenes. Vis Comput 21(8-10):629–638CrossRefGoogle Scholar
  55. Wu W, Arefin MA, Huang Z, Agarwal P, Shi S, Rivas R, Nahrstedt K (2010) “I’m the Jedi!”—a case study of user experience in 3D tele-immersive gaming. In: ISM, pp 220–227Google Scholar
  56. Wu W, Arefin A, Kurillo G, Agarwal P, Nahrstedt K, Bajcsy R (2011) Color-plus-depth level-of-details in 3D teleimmersive video—a psychophysical approach. In: Proceedings of ACM multimediaGoogle Scholar
  57. Yang Y, Wang X, Chen JX (2002) Rendering avatars in virtual reality: integrating a 3D model with 2D images. Comput Sci Eng 4(1):86–91MathSciNetCrossRefGoogle Scholar
  58. Wu W, Arefin A, Rivas R, Nahrstedt K, Sheppard R, Yang Z (2009) Quality of experience in distributed interactive multimedia environments: toward a theoretical framework. In: Proceedings of ACM multimedia, Beijing, pp 481-490Google Scholar
  59. Yang Z, Nahrstedt K, Cui Y, Yu B, Liang J, Jung SH, Bajcsy R (2005) TEEVE: The next generation architecture for tele-immersive environment. In: Seventh IEEE international symposium on multimedia (ISM 2005). IEEE Computer Society, Irvine, pp 112–119Google Scholar
  60. Yang Z, Wu W, Nahrstedt K, Kurillo G, Bajcsy R (2010) Enabling multi-party 3D tele-immersive environments with viewcast. ACM transactions on multimedia computing, communications, and applications (TOMCCAP) 6:1–30Google Scholar
  61. Yang Z, Wu W, Nahrstedt K, Kurillo G, Bajscy R (2007) ViewCast: view dissemination and management for multi-party 3D tele-immersive environments. In: Proceedings of ACM multimedia, Augsburg, pp 882–891Google Scholar
  62. Zhang D, Nomura Y, Fujii S (1991) Error analysis and optimization of camera calibration. In: Proceedings of IEEE/RSJ international workshop on intelligent robots and systems (IROS 91), Osaka, pp 292–296Google Scholar

Copyright information

© Springer-Verlag London 2012

Authors and Affiliations

  1. 1.Department of Electrical Engineering and Computer SciencesUniversity of California, BerkeleyBerkeleyUSA

Personalised recommendations