Multimedia Tools and Applications

, Volume 75, Issue 20, pp 12351–12363 | Cite as

Augmented immersion: video cutout and gesture-guided embedding for gaming applications

Article
First Online:
Received:
Revised:
Accepted:
  • 191 Downloads

Abstract

This paper introduces a novel framework, Gesture and Appearance Cutout Embedding (GACE), that supports real-time integration of human appearance and gesture-guided control within a game. It aims to enhance immersion since it allows game users to see their personal appearance in a real-time manner. In addition, we exploit the gesture-based control to allow user interaction with other in-game characters. With the goal to make implementation easier, we address the challenges in the whole pipeline of video processing, gesture recognition, and communication. The system is successfully integrated into both entertainment and serious games. Extensive experiments show that the proposed system runs reliably and comfortably with a commodity setting. Meanwhile, the user impression study indicates our system is favored by end users.

Keywords

Immersive system Human computer interaction Gaming applications 
This is a preview of subscription content, log in to check access.

Notes

Acknowledgments

This work was supported by Singapore Ministry of Education under research Grant MOE2012-TIF-2-G-016.

References

  1. 1.
    Aggarwal J, Ryoo M (2011) Human activity analysis: A review. ACM Comput Surv 43(3):16CrossRefGoogle Scholar
  2. 2.
    Cavazza M, Charles F, Mead SJ, Martin O., Marichal X., Nandi A. (2004) Multimodal acting in mixed reality interactive storytelling. IEEE MultiMedia 11(3):30–39CrossRefGoogle Scholar
  3. 3.
    Chambel T, Viana P, Bove VM, Strover S, Thomas G (2014) Immersiveme’14: 2nd ACM international workshop on immersive media experiences. In: ACM Multimedia, pp 1255–1256Google Scholar
  4. 4.
    Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: IEEE Conference on computer vision and pattern recognition, pp 886–893Google Scholar
  5. 5.
    Dede C (2009) Immersive interfaces for engagement and learning. Science 323 (5910):66–69CrossRefGoogle Scholar
  6. 6.
    Dondlinger MJ (2007) Educational video game design: A review of the literature. Journal of Applied Educational Technology 4(1):21–31Google Scholar
  7. 7.
    Henry P, Krainin M, Herbst E, Ren X, Fox D (2010) Rgb-d mapping: Using depth cameras for dense 3d modeling of indoor environments. In: International symposium on experimental robotics, pp 477–491Google Scholar
  8. 8.
    Izadi S, Kim D, Hilliges O, Molyneaux D, Newcombe RA, Kohli P, Shotton J, Hodges S, Freeman D, Davison AJ, Fitzgibbon AW (2011) Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In: ACM Symposium on User Interface Software and Technology, pp 559–568Google Scholar
  9. 9.
    Jörg S, Normoyle A, Safonova A (2012) How responsiveness affects players’ perception in digital games. In: Proceedings of the ACM Symposium on Applied Perception. ACM, pp 33–38Google Scholar
  10. 10.
    Lai K, Bo L, Ren X, Fox D (2011) Sparse distance learning for object recognition combining rgb and depth information. In: IEEE International Conference on Robotics and Automation, pp 4007–4013Google Scholar
  11. 11.
    Lang C, Nguyen TV, Katti H, Yadati K., Kankanhalli MS, Yan S (2012) Depth matters: Influence of depth cues on visual saliency. In: European conference on computer vision (2), pp 101–115Google Scholar
  12. 12.
    Lange B, Suma EA, Newman B, Phan T, Chang C-Y, Rizzo A, Bolas M (2011) Leveraging unencumbered full body control of animated virtual characters for game-based rehabilitation. In: International conference, virtual and mixed reality, pp 243–252Google Scholar
  13. 13.
    Likert R (1932) A technique for the measurement of attitudes. Archives of PsychologyGoogle Scholar
  14. 14.
    Liu S, Nguyen TV, Feng J, Wang M, Yan S (2012) Hi, magic closet, tell me what to wear!. In: ACM Multimedia, pp 1333–1334Google Scholar
  15. 15.
    Lu J, Nguyen V, Niu Z, Singh B, Luo Z, Do M (2011) Cutechat: a lightweight tele-immersive video chat system. In: ACM Multimedia, pp 1309–1312Google Scholar
  16. 16.
    Nguyen TV, Feng J, Yan S (2014) Seeing human weight from a single rgb-d image. J Comput Sci Technol 29(5):777–784CrossRefGoogle Scholar
  17. 17.
    Nguyen TV, Li L, Tan J, Yan S (2012) 3DME: 3d media express from rgb-d images. In: ACM Multimedia, pp 1331–1332Google Scholar
  18. 18.
    Nguyen TV, Tan YH, Sepulveda J (2014) GACE: Gesture and appearance cutout embedding for gaming applications. In: Proceedings of the 2nd ACM International Workshop on Immersive Media Experiences, ImmersiveMe ’14, pp 41–44. ACM, New YorkGoogle Scholar
  19. 19.
    Shotton J, Fitzgibbon AW, Cook M, Sharp T, Finocchio M, Moore R, Kipman A, Blake A (2011) Real-time human pose recognition in parts from single depth images. In: IEEE Conference on computer vision and pattern recognition, pp 1297–1304Google Scholar
  20. 20.
    Slater M, Spanlang B, Sanchez-Vives MV, Blanke O et al (2010) First person experience of body transfer in virtual reality. PloS one 5(5):e10564CrossRefGoogle Scholar
  21. 21.
    Soltani F, Eskandari F, Golestan S (2012) Developing a gesture-based game for deaf/mute people using microsoft kinect. In: International Conference on Complex, Intelligent, and Software Intensive Systems, pp 491–495Google Scholar
  22. 22.
    Tang S, Wang X, Lv X, Han TX, Keller JM, He Z, Skubic M, Lao S (2012) Histogram of oriented normal vectors for object recognition with a depth sensor. In: Asian conference on computer vision, pp 525–538Google Scholar
  23. 23.
    Wang J, Liu Z, Wu Y, Yuan J (2012) Mining actionlet ensemble for action recognition with depth cameras. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 1290–1297Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.ARTIC CentreSingapore PolytechnicSingaporeSingapore

Personalised recommendations