Natural Interaction with Video Environments Using Gestures and a Mirror Image Avatar

  • Christian Kray
  • Dennis Wilhelm
  • Thore Fechner
  • Morin Ostkmap
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9297)


Video environments are a promising option for a variety of applications such as training, gaming, entertainment, remote collaboration, or user studies. Being able to interact with these environments enables further applications and extends existing application scenarios. In this paper, we propose a novel interaction technique that combines natural gestures with mirror images of the user to allow for immersive interaction with video environments. The technique enables movement inside the 3D space depicted by the video as well as the placement and manipulation of virtual objects within the 3D space. We describe a potential application scenario, where interactive public displays are placed inside a scene by one user and then experienced by another user. We also briefly report on a user study evaluating the gesture set we defined for controlling movement within the video.


Gestural interaction Mirror image Avatar Video 


  1. 1.
    Ahn, S.C., Lee, T.-S., Kim, I.-J., Kwon, Y.-M., Kim, H.-G.: Large display interaction using video avatar and hand gesture recognition. In: Campilho, A.C., Kamel, M.S. (eds.) ICIAR 2004. LNCS, vol. 3211, pp. 261–268. Springer, Heidelberg (2004) CrossRefGoogle Scholar
  2. 2.
    Ballendat, T., Marquardt, N., Greenberg, S.: Proxemic interaction: designing for a proximity and orientation-aware environment. In: Proceedings of ITS 2010, pp. 121–130. ACM (2010)Google Scholar
  3. 3.
    Benko, H., Wilson, A.D.: Multi-point interactions with immersive omnidirectional visualizations in a dome. In: Proceedings of ITS 2010, pp. 19–28. ACM (2010)Google Scholar
  4. 4.
    Gotardo, P.F.U., Price, A.: Integrated space: authoring in an immersive environment with 3D body tracking. In: SIGGRAPH 2010 Posters. ACM (2010)Google Scholar
  5. 5.
    Iwabuchi, E., Nakagawa, M., Siio, I.: Smart makeup mirror: computer-augmented mirror to aid makeup application. In: Jacko, J.A. (ed.) HCI International 2009, Part IV. LNCS, vol. 5613, pp. 495–503. Springer, Heidelberg (2009) Google Scholar
  6. 6.
    Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A., Fitzgibbon, A.: Kinectfusion: real-time 3D reconstruction and interaction using a moving depth camera. In: Proceedings of UIST 2011, pp. 559–568. ACM (2011)Google Scholar
  7. 7.
    Morikawa, O., Maesako, T.: Hypermirror: toward pleasant-to-use video mediated communication system. In: Proceedings of CSCW 1998, pp. 149–158. ACM (1998)Google Scholar
  8. 8.
    Nancel, M., Wagner, J., Pietriga, E., Chapuis, O., Mackay, W.: Mid-air pan-and-zoom on wall-sized displays. In: Proceedings of CHI 2011, pp. 177–186. ACM (2011)Google Scholar
  9. 9.
    Särkelä, H., Takatalo, J., May, P., Laakso, M., Nyman, G.: The movement patterns and the experiential components of virtual environments. Int. J. Hum.-Comput. Stud. 67(9), 787–799 (2009)zbMATHCrossRefGoogle Scholar
  10. 10.
    Andrés del Valle, A.C., Opalach, A.: The persuasive mirror: computerized persuasion for healthy living. In: Proceedings of HCI International 2005 (2005)Google Scholar
  11. 11.
    Vera, L., Gimeno, J., Coma, I., Fernández, M.: Augmented mirror: interactive augmented reality system based on kinect. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) INTERACT 2011, Part IV. LNCS, vol. 6949, pp. 483–486. Springer, Heidelberg (2011) CrossRefGoogle Scholar
  12. 12.
    Vogel, D., Balakrishnan, R.: Distant freehand pointing and clicking on very large, high resolution displays. In: Proceedings of UIST 2005, pp. 33–42. ACM (2005)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2015

Authors and Affiliations

  • Christian Kray
    • 1
  • Dennis Wilhelm
    • 1
  • Thore Fechner
    • 1
  • Morin Ostkmap
    • 1
  1. 1.Institute for GeoinformaticsWestfälische Wilhelms-Universität MünsterMünsterGermany

Personalised recommendations