Multimedia Tools and Applications

, Volume 73, Issue 1, pp 491–517 | Cite as

Gesture control interface for immersive panoramic displays

  • Marcel Alcoverro
  • Xavier SuauEmail author
  • Josep R. Morros
  • Adolfo López-Méndez
  • Albert Gil
  • Javier Ruiz-Hidalgo
  • Josep R. Casas


In this paper, we propose a gesture-based interface designed to interact with panoramic scenes. The system combines novel static gestures with a fast hand tracking method. Our proposal is to use static gestures as shortcuts to activate functionalities of the system (i.e. volume up/down, mute, pause, etc.), and hand tracking to freely explore the panoramic video. The overall system is multi-user, and incorporates a user identification module based on face recognition, which is able both to recognize returning users and to add new users online. The system exploits depth data, making it robust to challenging illumination conditions. We show through experimental results the performance of every component of the system compared to the state of the art. We also show the results of a usability study performed with several untrained users.


Interactivity Panoramic display Human-machine interfaces 


  1. 1.
    Ahonen T, Hadid A, Pietikainen M (2006) Face description with local binary patterns: application to face recognition. IEEE Trans Pattern Anal Mach Intell 28(12):2037–2041. doi: 10.1109/TPAMI.2006.244 CrossRefGoogle Scholar
  2. 2.
    Bhuiyan M, Picking R (2011) A gesture controlled user interface for inclusive design and evaluative study of its usability. J Softw Eng Appl 4(9):513–521CrossRefGoogle Scholar
  3. 3.
    Bradski G (2000) The OpenCV library. Dr. Dobb’s journal of software toolsGoogle Scholar
  4. 4.
    Breiman L (2001) Random forests. Mach Learn 45(1):5–32Google Scholar
  5. 5.
    Demirdjian D, Varri C (2009) Recognizing events with temporal random forests. In: Proceedings of the 2009 international conference on multimodal interfaces, ICMI-MLMI ’09. ACM, New York, pp 293–296. doi: 10.1145/1647314.1647377
  6. 6.
    Duda R, Hart P, Stork D (2001) Pattern classification, 2nd edn. Wiley, New YorkzbMATHGoogle Scholar
  7. 7.
    Fascinate: Format-agnostic script-based interactive experience. Accessed 28 Feb 2013
  8. 8.
    Francese R, Passero I, Tortora G (2012) Wiimote and kinect: gestural user interfaces add a natural third dimension to hci. In: Proceedings of the international working conference on advanced visual interfaces, AVI ’12. ACM, New York, pp 116–123. doi: 10.1145/2254556.2254580
  9. 9.
    Friedman JH, Bentley JL, Finkel RA (1977) An algorithm for finding best matches in logarithmic expected time. Trans Math Softw 3(3):209–226. doi: 10.1145/355744.355745 CrossRefzbMATHGoogle Scholar
  10. 10.
    Gall J, Yao A, Razavi N, Van Gool L, Lempitsky V (2011) Hough forests for object detection, tracking, and action recognition. IEEE Trans Pattern Anal Mach Intell 33(11):2188–2202. doi: 10.1109/TPAMI.2011.70 CrossRefGoogle Scholar
  11. 11.
    Gesturepak: Gesture recording and recognition toolkit. Accessed 20 Feb 2013
  12. 12.
    Jaimes A, Sebe N (2007) Multimodal human–computer interaction: a survey. Comput Vis Image Underst 108(1–2):116–134. doi: 10.1016/j.cviu.2006.10.019 (special issue on vision for human-computer interaction)Google Scholar
  13. 13.
    Ji Q, Wechsler H, Duchowski A, Flickner M (2005) Editorial: special issue: eye detection and tracking. Comput. Vis. Image Underst. 98(1):1–3. doi: 10.1016/j.cviu.2004.07.006 CrossRefGoogle Scholar
  14. 14.
    Kinect for windows sdk. Accessed 20 Feb 2013
  15. 15.
    Lausberg H, Sloetjes H (2009) Coding gestural behavior with the neuroges-elan system. Behav Res Methods 41(3):841–849CrossRefGoogle Scholar
  16. 16.
    Liu J, Zhong L, Wickramasuriya J, Vasudevan V (2009) uwave: accelerometer-based personalized gesture recognition and its applications. Pervasive Mobile Comput 5(6):657–675. doi: 10.1016/j.pmcj.2009.07.007 (PerCom 2009)
  17. 17.
    López-Méndez A, Casas JR (2012) Can our tv robustly understand human gestures?: real-time gesture localization in range data. In: Proceedings of the 9th European conference on visual media production, CVMP ’12. ACM, New York, pp 18–25. doi: 10.1145/2414688.2414691
  18. 18.
    Nielsen M, Störring M, Moeslund T, Granum E (2004) A procedure for developing intuitive and ergonomic gesture interfaces for hci. In: Camurri A, Volpe G (eds) Gesture-based communication in human-computer interaction. Lecture notes in computer science, vol 2915, pp 409–420. Springer Berlin HeidelbergGoogle Scholar
  19. 19.
    Norman DA (2010) Natural user interfaces are not natural. Interactions 17(3):6–10CrossRefGoogle Scholar
  20. 20.
    Nui Group Community. Accessed 23 Feb 2013
  21. 21.
    Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on featured distributions. Patt Recogn 29(1):51–59CrossRefGoogle Scholar
  22. 22.
    Openni sdk. Accessed 20 Feb 2013
  23. 23.
    Pantic M, Rothkrantz LJM (2000) Automatic analysis of facial expressions: the state of the art. IEEE Trans Pattern Anal Mach Intell 22(12):1424–1445. doi: 10.1109/34.895976 CrossRefGoogle Scholar
  24. 24.
  25. 25.
    Poppe R: Vision-based human motion analysis: an overview. Comp Vision Image Underst 108(1–2):4–18 (2007). doi: 10.1016/j.cviu.2006.10.016 (special issue on vision for human-computer interaction)Google Scholar
  26. 26.
    Potamianos G, Neti C, Luettin J, Matthews I (2004) Audio-visual automatic speech recognition: An overview. Issues in Visual and Audio-Visual Speech Processing, pp 356–396Google Scholar
  27. 27.
    Pugeault N, Bowden R (2011) Spelling it out: real-time ASL fingerspelling recognition. In: ICCV-CDC4CVGoogle Scholar
  28. 28.
    Ren Z, Yuan J, Zhang Z (2011) Robust hand gesture recognition based on finger-earth mover’s distance with a commodity depth camera. In: ACM MM, MM ’11. ACM, New York, pp 1093–1096. doi: 10.1145/2072298.2071946
  29. 29.
    Schlömer T, Poppinga B, Henze N, Boll S (2008) Gesture recognition with a wii controller. In: Proceedings of the 2nd international conference on tangible and embedded interaction, TEI ’08. ACM, New York, pp 11–14. doi: 10.1145/1347390.1347395
  30. 30.
    Sebe N (2009) Multimodal interfaces: challenges and perspectives. J Ambient Intell Smart Environ 1(1):23–30Google Scholar
  31. 31.
    Shotton J, Fitzgibbon A, Cook M, Sharp T, Finocchio M, Moore R, Kipman A, Blake A (2011) Real-time human pose recognition in parts from single depth images. In: CVPR, pp 1297–1304. doi: 10.1109/CVPR.2011.5995316
  32. 32.
    Stern HI, Wachs JP, Edan Y (2008) Designing hand gesture vocabularies for natural interaction by combining psycho-physiological and recognition factors. Int J Semantic Comput 02(01):137–160. doi: 10.1142/S1793351X08000385 CrossRefGoogle Scholar
  33. 33.
    Suau X, Ruiz-Hidalgo J, Casas JR (2012) Real-time head and hand tracking based on 2.5D data. Trans Multimed 1(99):1Google Scholar
  34. 34.
    Turk M (2001) Gesture recognition. Handbook of Virtual Environment TechnologyGoogle Scholar
  35. 35.
    Uebersax D, Gall J, Van den Bergh M, Van Gool L (2011) Real-time sign language letter and word recognition from depth data. In: ICCV-HCI, pp 1–8Google Scholar
  36. 36.
    Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vision 57(2):137–154CrossRefGoogle Scholar
  37. 37.
    Wachs JP, Kölsch M, Stern H, Edan Y (2011) Vision-based hand-gesture applications. Commun ACM 54(2):60–71CrossRefGoogle Scholar
  38. 38.
    Zhao W, Chellappa R, Phillips PJ, Rosenfeld A (2003) Face recognition: a literature survey. ACM Comput Surv 35(4):399–458. doi: 10.1145/954339.954342 CrossRefGoogle Scholar
  39. 39.
    Zigfu. Motion controlled web. Accessed 20 Feb 2013

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Marcel Alcoverro
    • 1
  • Xavier Suau
    • 1
    Email author
  • Josep R. Morros
    • 1
  • Adolfo López-Méndez
    • 1
  • Albert Gil
    • 1
  • Javier Ruiz-Hidalgo
    • 1
  • Josep R. Casas
    • 1
  1. 1.Department of Signal Theory and CommunicationsUniversitat Politècnica de CatalunyaBarcelonaSpain

Personalised recommendations