Advertisement

Low-bandwidth 3D visual telepresence system

  • Diana-Margarita Córdova-EsparzaEmail author
  • Juan R. Terven
  • Hugo Jiménez-Hernández
  • Ana Herrera-Navarro
  • Alberto Vázquez-Cervantes
  • Juan-M. García-Huerta
Article
  • 24 Downloads

Abstract

We present a methodology to develop a low-cost, low-bandwidth visual telepresence system using commodity depth sensors. To obtain a precise representation of the participants, we fuse together multiple views extracted using a deep background subtraction method. We build a proof-of-concept display composed of a video projector and a quadrangular pyramid made of acrylic, to demonstrate the visualization of a remote person without the need for head-mounted displays. Our system represents an attempt to democratize high-fidelity 3D telepresence using off-the-shelf components.

Keywords

Telepresence Holographic display 3D reconstruction RGB-D cameras Background subtraction 

Notes

Acknowledgements

This work was supported by CONACYT through postdoctoral support number 291113. We also want to thank CIDESI for providing the facilities and assistance during the development of this project.

Compliance with Ethical Standards

Conflict of interests

The authors declare that there is no conflict of interest.

References

  1. 1.
    Antonio S, Herrera R, Enriquez E (2013) Projection’s panel of models for touch screen. Int J Innovative Res Comput Commun Eng 1(9):2057–2064Google Scholar
  2. 2.
    Babaee M, Dinh DT, Rigoll G (2018) A deep convolutional neural network for video sequence background subtraction. Pattern Recogn 76:635–649CrossRefGoogle Scholar
  3. 3.
    Badrinarayanan V, Kendall A, Cipolla R (2015) Segnet: a deep convolutional encoder-decoder architecture for image segmentation. arXiv:1511.00561
  4. 4.
    Beck S, Kunert A, Kulik A, Froehlich B (2013) Immersive group-to-group telepresence. IEEE Trans Vis Comput Graph 19(4):616–625CrossRefGoogle Scholar
  5. 5.
    Bimber O, Raskar R (2005) Spatial augmented reality: merging real and virtual worlds. AK Peters/ CRC PressGoogle Scholar
  6. 6.
    Blanche PA, Bablumian A, Voorakaranam R, Christenson C, Lin W, Gu T, Flores D, Wang P, Hsieh WY, Kathaperumal M et al (2010) Holographic three-dimensional telepresence using large-area photorefractive polymer. Nature 468(7320):80CrossRefGoogle Scholar
  7. 7.
    Brown DC (1971) Lens distortion for close-range photogrammetry. Photogramm Eng 37(8):855–866Google Scholar
  8. 8.
    Córdova-Esparza DM, Terven JR, Jiménez-Hernández H, Herrera-Navarro AM (2017) A multiple camera calibration and point cloud fusion tool for Kinect V2. Sci Comput Program 143:1–8CrossRefGoogle Scholar
  9. 9.
    Dalvi AA, Siddavatam I, Dandekar NG, Patil AV (2015) 3D holographic projections using prism and hand gesture recognition. In: Proceedings of the 2015 international conference on advanced research in computer science engineering & technology (ICARCSET 2015). ACM, p 18Google Scholar
  10. 10.
    Dreshaj E (2015) Holosuite: an exploration into interactive holographic telepresence (Doctoral dissertation Massachusetts Institute of Technology)Google Scholar
  11. 11.
    Duane CB (1971) Close-range camera calibration. Photogramm Eng 37(8):855–866Google Scholar
  12. 12.
    Eigen D, Fergus R (2015) Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In: Proceedings of the IEEE international conference on computer vision, pp 2650–2658Google Scholar
  13. 13.
    Gohane MST, Longadge MRN (2014) 3D holograph projection-future of visual communicationGoogle Scholar
  14. 14.
    Goyette N, Jodoin PM, Porikli F, Konrad J, Ishwar P (2012) Changedetection. net: a new change detection benchmark dataset. In: 2012 IEEE computer society conference on computer vision and pattern recognition workshops. IEEE, pp 1–8Google Scholar
  15. 15.
    Hofmann M, Tiefenbacher P, Rigoll G (2012) Background segmentation with feedback: the pixel-based adaptive segmenter. In: 2012 IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW). IEEE, pp 38–43Google Scholar
  16. 16.
    Lee H, Ha G, Lee S, Cha J, Kim S (2016) A hologram based tele-existence platform for emotional exchange among a group of users in both real and virtual environments. In: Proceedings of the 22nd ACM conference on virtual reality software and technology. ACM, pp 295–296Google Scholar
  17. 17.
    Lee H, Ha G, Lee S, Kim S (2017) A mixed reality tele-presence platform to exchange emotion and sensory information based on MPEG-V standard. In: 2017 IEEE Virtual Reality (VR). IEEE, pp 349–350Google Scholar
  18. 18.
    Levenberg K (1944) A method for the solution of certain non-linear problems in least squares. Q Appl Math 2(2):164–168MathSciNetCrossRefGoogle Scholar
  19. 19.
    Lu X, Shen J, Perugini S, Yang J (2015) An immersive telepresence system using rgb-d sensors and head mounted display. In: 2015 IEEE international symposium on multimedia (ISM). IEEE, pp 453–458Google Scholar
  20. 20.
    Maimone A, Fuchs H (2011) Encumbrance-free telepresence system with real-time 3D capture and display using commodity depth cameras. In: 2011 10th IEEE international symposium on mixed and augmented reality (ISMAR). IEEE, pp 137–146Google Scholar
  21. 21.
    Maimone A, Fuchs H (2012) Real-time volumetric 3D capture of room-sized scenes for telepresence. In: 3DTV-Conference: the true vision-capture, transmission and display of 3D video (3DTV-CON). IEEE, pp 1–4Google Scholar
  22. 22.
    Maimone A, Yang X, Dierk N, State A, Dou M, Fuchs H (2013) General-purpose telepresence with head-worn optical see-through displays and projector-based lighting. In: 2013 IEEE virtual reality (VR). IEEE, pp 23–26Google Scholar
  23. 23.
    Marquardt DW (1963) An algorithm for least-squares estimation of nonlinear parameters. J Soc Ind Appl Math 11(2):431–441MathSciNetCrossRefGoogle Scholar
  24. 24.
    Mekuria R, Cesar P (2016) MP3DG-PCC, open source software framework for implementation and evaluation of point cloud compression. In: Proceedings of the 24th ACM international conference on multimedia. ACM, pp 1222–1226Google Scholar
  25. 25.
    Noor AK, Aras R (2015) Potential of multimodal and multiuser interaction with virtual holography. Adv Eng Softw 81:1–6CrossRefGoogle Scholar
  26. 26.
    Orts-Escolano S, Rhemann C, Fanello S, Chang W, Kowdle A, Degtyarev Y, Kim D, Davidson PL, Khamis S, Dou M et al (2016) Holoportation: virtual 3d teleportation in real-time. In: Proceedings of the 29th annual symposium on user interface software and technology. ACM, pp 741–754Google Scholar
  27. 27.
    Pejsa T, Kantor J, Benko H, Ofek E, Wilson A (2016) Room2room: enabling life-size telepresence in a projected augmented reality environment. In: Proceedings of the 19th ACM conference on computer-supported cooperative work & social computing. ACM, pp 1716–1725Google Scholar
  28. 28.
    Prince SJ (2012) Computer vision: models, learning, and inference. Cambridge University Press, United KingdomCrossRefGoogle Scholar
  29. 29.
    Sajid H, Cheung SCS (2015) Background subtraction for static & moving camera. In: 2015 IEEE international conference on image processing (ICIP). IEEE, pp 4530–4534Google Scholar
  30. 30.
    Sakkos D, Liu H, Han J, Shao L (2017) End-to-end video background subtraction with 3d convolutional neural networks. Multimed Tools Appl, pp 1–19Google Scholar
  31. 31.
    Sedky M, Chibelushi CC, MONIRI M (2010) Image processing: object segmentation using full-spectrum matching of albedo derived from colour imagesGoogle Scholar
  32. 32.
    Schwarz S, Preda M, Baroncini V, Budagavi M, Cesar P, Chou PA, Llach J (2018) Emerging MPEG standards for point cloud compression. IEEE J Emerging Sel Top Circuits Syst 9(1):133–148CrossRefGoogle Scholar
  33. 33.
    St-Charles PL, Bilodeau GA, Bergevin R (2015) A self-adjusting approach to change detection based on background word consensus. In: 2015 IEEE winter conference on applications of computer vision (WACV). IEEE, pp 990–997Google Scholar
  34. 34.
    St-Charles PL, Bilodeau GA, Bergevin R (2015) Subsense: a universal change detection method with local adaptive sensitivity. IEEE Trans Image Process 24(1):359–373MathSciNetCrossRefGoogle Scholar
  35. 35.
    Statista: global business travel spending in 2015, 2016 and 2020 (in trillion U.S. dollars) (2017). https://www.statista.com/statistics/612244/global-business-travel-spending/ [Online; accessed 1-April-2018]
  36. 36.
    Stauffer C, Grimson WEL (1999) Adaptive background mixture models for real-time tracking. In: CVPR. IEEE, p 2246Google Scholar
  37. 37.
    Terven JR, Córdova-Esparza DM (2016) Kin2. A Kinect 2 toolbox for MATLAB. Sci Comput Program 130:97–106CrossRefGoogle Scholar
  38. 38.
    Tiro D, Poturiović A, Buzadjija N (2015) The possibility of the hologram pyramid applying in the rapid prototyping. In: 2015 4th mediterranean conference on embedded computing (MECO). IEEE, pp 25–30Google Scholar
  39. 39.
    Towles H, Chen WC, Yang R, Kum SU, Kelshikar HFN, Mulligan J, Daniilidis K, Fuchs H, Hill CC, Mulligan NKJ et al (2002) 3D tele-collaboration over internet2. International Workshop on Immersive Telepresence, Juan Les PinsGoogle Scholar
  40. 40.
    Varadarajan S, Miller P, Zhou H (2015) Region-based mixture of gaussians modelling for foreground detection in dynamic scenes. Pattern Recogn 48(11):3488–3503CrossRefGoogle Scholar
  41. 41.
    Wikimedia Commons. File:ABC Clarke predicts internet and PC.ogv — Wikimedia Commons{,} the free media repository (2018). https://commons.wikimedia.org/w/index.php?title=File:ABC_Clarke_predicts_internet_and_PC.ogv&oldid=292577968 [Online; accessed 1-April-2018]
  42. 42.
    Yoo H, Kim H (2014) On study of the volumetric display techniques. In: Interactive media arts proceedingsGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.UAQ, Universidad Autónoma de Querétaro, Facultad de InformáticaQuerétaroMéxico
  2. 2.AiFi Inc.Santa ClaraUSA
  3. 3.CIDESI, Centro de Ingeniería y Desarrollo IndustrialQuerétaroMéxico

Personalised recommendations