The Visual Computer

, 26:3 | Cite as

Multi-camera tele-immersion system with real-time model driven data compression

A new model-based compression method for massive dynamic point data
  • Jyh-Ming LienEmail author
  • Gregorij Kurillo
  • Ruzena Bajcsy
Original Article


Vision-based full-body 3D reconstruction for tele-immersive applications generates large amount of data points, which have to be sent through the network in real time. In this paper, we introduce a skeleton-based compression method using motion estimation where kinematic parameters of the human body are extracted from the point cloud data in each frame. First we address the issues regarding the data capturing and transfer to a remote site for the tele-immersive collaboration. We compare the results of the existing compression methods and the proposed skeleton-based compression technique. We examine the robustness and efficiency of the algorithm through experimental results with our multi-camera tele-immersion system. The proposed skeleton-based method provides high and flexible compression ratios from 50:1 to 5000:1 with reasonable reconstruction quality (peak signal-to-noise ratio from 28 to 31 dB) while preserving real-time (10+ fps) processing.


3D tele-immersion Multi-camera tele-immersion system Point data compression Model-based compression 


  1. 1.
    Adobe. Quicktime 7.0 h.264 implementation (2006) Google Scholar
  2. 2.
    Aggarwal, J.K., Cai, Q.: Human motion analysis: a review. Comput. Vis. Image Underst. 73(3), 428–440 (1999) CrossRefGoogle Scholar
  3. 3.
    Arikan, O.: Compression of motion capture databases. ACM Trans. Graph. 25(3), 890–897 (2006) CrossRefMathSciNetGoogle Scholar
  4. 4.
    Baker, H., Tanguay, D., Sobel, I., Gelb, D., Gross, M., Culbertson, W., Malzenbender, T.: The coliseum immersive teleconferencing system. In: Proceedings of International Workshop on Immersive Telepresence, Juan-les-Pins, France (2002) Google Scholar
  5. 5.
    Bentley, J.L.: Multidimensional binary search trees used for associative searching. Commun. ACM 18(9), 509–517 (1975) zbMATHCrossRefMathSciNetGoogle Scholar
  6. 6.
    Besl, P.J., McKay, N.D.: A method for registration of 3-D shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 239–256 (1992) CrossRefGoogle Scholar
  7. 7.
    Cheung, K.M., Baker, S., Kanade, T.: Shape-from-silhouette of articulated objects and its use for human body kinematics estimation and motion capture. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 2003 Google Scholar
  8. 8.
    CMU. Graphics Lab Motion Capture Database. Carnegie Mellon University.
  9. 9.
    Demirdjian, D., Darrell, T.: 3-D articulated pose tracking for untethered deictic reference. In: ICMI’02: Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, Washington, DC, p. 267. IEEE Comput. Soc., Los Alamitos (2002) CrossRefGoogle Scholar
  10. 10.
    Dewaele, G., Devernay, F., Horaud, R.: Hand motion from 3D point trajectories and a smooth surface model. In: ECCV (1), pp. 495–507 (2004) Google Scholar
  11. 11.
    Gavrila, D.M.: The visual analysis of human movement: a survey. Comput. Vis. Image Underst. 73(1), 82–98 (1999) zbMATHCrossRefGoogle Scholar
  12. 12.
    Gross, M., Würmlin, S., Naef, M., Lamboray, E., Spagno, C., Kunz, A., Koller-Meier, E., Svoboda, T., Gool, L.V., Lang, S., Strehlke, K., Moere, A.V., Staadt, O.: Blue-c: a spatially immersive display and 3D video portal for telepresence. ACM Trans. Graph. 22(3), 819–827 (2003) CrossRefGoogle Scholar
  13. 13.
    Gumhold, S., Karni, Z., Isenburg, M., Seidel, H.-P.: Predictive point-cloud compression. In: Siggraph 2005 Sketches (2005) Google Scholar
  14. 14.
    Hasenfratz, J., Lapierre, M., Sillion, F.: A real-time system for full-body interaction with virtual worlds. In: Proceedings of Eurographics Symposium on Virtual Environments, pp. 147–156. Eurographics Association, Aire-la-Ville (2004) Google Scholar
  15. 15.
    Holden, M.K.: Virtual environments for motor rehabilitation: review. Cyberpsychol. Behav. 8(3), 187–211 (2005) CrossRefMathSciNetGoogle Scholar
  16. 16.
    Ibarria, L., Rossignac, J.: Dynapack: space-time compression of the 3D animations of triangle meshes with fixed connectivity. In: SCA’03: Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Aire-la-Ville, Switzerland, pp. 126–135. Eurographics Association, Aire-la-Ville (2003) Google Scholar
  17. 17.
    Isard, M., Blake, A.: Condensation—conditional density propagation for visual tracking. Int. J. Comput. Vis. 29(1), 5–28 (1998) CrossRefGoogle Scholar
  18. 18.
    Jung, S., Bajcsy, R.: A framework for constructing real-time immersive environments for training physical activities. J. Multimed. 1(7), 9–17 (2006) Google Scholar
  19. 19.
    Kalra, P., Magnenat-Thalman, N., Moccozet, L., Sannier, G., Aubel, A., Thalman, D.: Real-time animation of realistic virtual humans. IEEE Comput. Graph. Appl. 18(25), 42–56 (1998) CrossRefGoogle Scholar
  20. 20.
    Karni, Z., Gotsman, C.: Compression of soft-body animation sequences. Comput. Graph. 28(1), 25–34 (2004) CrossRefGoogle Scholar
  21. 21.
    Keshner, E.A.: Virtual reality and physical rehabilitation: a new toy or a new research and rehabilitation tool? J. Neuroengineering Rehabil. 1(8) (2004) Google Scholar
  22. 22.
    Keshner, E., Kenyon, R.: Using immersive technology for postural research and rehabilitation. Assist. Technol. 16, 54–62 (2004) Google Scholar
  23. 23.
    Knoop, R.D.S., Vacek, S.: Sensor fusion for 3D human body tracking with an articulated 3D body model. In: Proceedings of the IEEE International Conference on Robotics and Automation, Walt Disney Resort, Orlando, Florida, 15 May 2006 Google Scholar
  24. 24.
    Kum, S.-U., Mayer-Patel, K.: Real-time multidepth stream compression. ACM Trans. Multimedia Comput. Commun. Appl. 1(2), 128–150 (2005) CrossRefGoogle Scholar
  25. 25.
    Lamboray, E., Würmlin, S., Gross, M.: Real-time streaming of point-based 3D video. In: VR’04: Proceedings of the IEEE Virtual Reality 2004, Washington, DC, p. 91. IEEE Comput. Soc., Los Alamitos (2004) CrossRefGoogle Scholar
  26. 26.
    Lanier, J.: Virtually there. Sci. Am. 4, 52–61 (2001) Google Scholar
  27. 27.
    Lengyel, J.E.: Compression of time-dependent geometry. In: SI3D’99: Proceedings of the 1999 Symposium on Interactive 3D Graphics, New York, NY, pp. 89–95. ACM, New York (1999) CrossRefGoogle Scholar
  28. 28.
    Li, L., Zhang, M., Xu, F., Liu, S.: Ert-vr: an immersive virtual reality system for emergency rescue training. Virtual Real. 8(3), 194–197 (2005) CrossRefGoogle Scholar
  29. 29.
    Lien, J.-M., Bajcsy, G.K.R.: Skeleton-based data compression for multi-camera tele-immersion system. In: Proceedings of the International Symposium on Visual Computing (ISVC), pp. 347–354 (2007) Google Scholar
  30. 30.
    Lopez, E.J.L.: Finite element surface-based stereo 3D reconstruction, April 2006. Poster presentation given at the Trust NSF Site Visit Google Scholar
  31. 31.
    Marc Alexa, W.M.: Representing animations by principal components. Comput. Graph. Forum 19(3), 411–418 (2000) CrossRefGoogle Scholar
  32. 32.
    Mason, H., Moutahir, M.: Multidisciplinary experiential education in second life: a global approach. In: Second Life Education Workshop, San Francisco, California, pp. 30–34 (2006) Google Scholar
  33. 33.
    McComas, J., Pivik, J., Laflamme, M.: Current uses of virtual reality for children with disabilities. Virtual Environments in Clinical Psychology and Neuroscience (1998) Google Scholar
  34. 34.
    Morency, L.-P., Darrell, T.: Stereo tracking using ICP and normal flow constraint. In: Proceedings of International Conference on Pattern Recognition (2002) Google Scholar
  35. 35.
    Mulligan, J., Daniilidis, K.: Real-time trinocular stereo for tele-immersion. In: Proceedings of 2001 International Conference on Image Processing, Thessaloniki, Greece, pp. 959–962 (2001) Google Scholar
  36. 36.
    Ochotta, T., Saupe, D.: Compression of point-based 3D models by shape-adaptive wavelet coding of multi-height fields. In: Symposium on Point-Based Graphics, pp. 103–112 (2004) Google Scholar
  37. 37.
    Patel, K., Bailenson, J.N., Hack-Jung, S., Diankov, R., Bajcsy, R.: The effects of fully immersive virtual reality on the learning of physical tasks. In: Proceedings of the 9th Annual International Workshop on Presence, Ohio, USA, pp. 87–94 (2006) Google Scholar
  38. 38.
    Piccardi, M.: Background subtraction techniques: a review. In: Proceedings of IEEE International Conference on Systems, Man and Cybernetics, Hague, Netherlands, pp. 3099–3104 (2004) Google Scholar
  39. 39.
    Point Grey Research Inc, Vancouver, Canada Google Scholar
  40. 40.
    Robb, R.: Virtual reality in medicine: A personal perspective. J. Vis. 5(4), 317–326 (2002) Google Scholar
  41. 41.
    Rusinkiewicz, S., Levoy, M.: Efficient variants of the ICP algorithm. In: Proceedings of the Third International Conference on 3-D Digital Imaging and Modeling (3DIM), pp. 145–152 (2001) Google Scholar
  42. 42.
    Sattler, M., Sarlette, R., Klein, R.: Simple and efficient compression of animation sequences. In: SCA’05: Proceedings of the 2005 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, New York, NY, pp. 209–217. ACM, New York (2005) CrossRefGoogle Scholar
  43. 43.
    Simon, D., Hebert, M., Kanade, T.: Real-time 3-D pose estimation using a high-speed range sensor. In: Proceedings of IEEE International Conference on Robotics and Automation (ICRA’94), vol. 3, pp. 2235–2241 (1994) Google Scholar
  44. 44.
    Tsai, R.: A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses. IEEE J. Robot. Autom. RA3(4), 323–344 (1987) CrossRefGoogle Scholar
  45. 45.
    William Pennebaker, J.M.: JPEG: Still Image Data Compression Standard. Springer, Berlin (1992) Google Scholar
  46. 46.
    Würmlin, S., Lamboray, E., Gross, M.: 3D video fragments: dynamic point samples for real-time free-viewpoint video. Comput. Graph. 28, 3–14 (2004) CrossRefGoogle Scholar
  47. 47.
    Yang, Y., Wang, X., Chen, J.X.: Rendering avatars in virtual reality: integrating a 3D model with 2D images. Comput. Sci. Eng. 4(1), 86–91 (2002) CrossRefMathSciNetGoogle Scholar
  48. 48.
    Yang, Z., Cui, Y., Anwar, Z., Bocchino, R., Kiyanclar, N., Nahrstedt, K., Campbell, R.H., Yurcik, W.: Real-time 3D video compression for tele-immersive environments. In: Proc. of SPIE/ACM Multimedia Computing and Networking (MMCN’06), San Jose, CA (2006) Google Scholar
  49. 49.
    Zhang, J., Owen, C.B.: Octree-based animated geometry compression. In: DCC’04: Proceedings of the Conference on Data Compression, Washington, DC, p. 508. IEEE Comput. Soc., Los Alamitos (2004) Google Scholar
  50. 50.
    Zhang, D., Nomura, Y., Fujii, S.: Error analysis and optimization of camera calibration. In: Proceedings of IEEE/RSJ International Workshop on Intelligent Robots and Systems (IROS’91), Osaka, Japan, pp. 292–296 (1991) Google Scholar
  51. 51.
    Zhao, W., Nandhakumar, N.: Effects of camera alignment errors on stereoscopic depth estimates. Pattern Recogn. 29(12), 2115–2126 (1996) CrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2009

Authors and Affiliations

  • Jyh-Ming Lien
    • 1
    Email author
  • Gregorij Kurillo
    • 2
  • Ruzena Bajcsy
    • 2
  1. 1.George Mason UniversityFairfaxUSA
  2. 2.University of CaliforniaBerkeleyUSA

Personalised recommendations