Multimedia Tools and Applications

, Volume 76, Issue 2, pp 1615–1638 | Cite as

A virtual mouse interface with a two-layered Bayesian network

  • Myung-Cheol Roh
  • Dongoh Kang
  • Sungju Huh
  • Seong-Whan Lee


During the last decade, many natural interaction methods between human and computer have been introduced. They were developed for substitutions of keyboard and mouse devices so that they provide convenient interfaces. Recently, many studies on vision based gestural control methods for Human-Computer Interaction (HCI) have been attracted attention because of their convenience and simpleness. Two of the key issues in these kinds of interfaces are robustness and real-time processing. This paper presents a hand gesture based virtual mouse interface and Two-layer Bayesian Network (TBN) for robust hand gesture recognition in real-time. The TBN provides an efficient framework to infer hand postures and gestures not only from information at the current time frame, but also from the preceding and following information, so that it compensates for erroneous postures and its locations under cluttered background environment. Experiments demonstrated that the proposed model recognized hand gestures with a recognition rate of 93.76 % and 85.15 % on simple and cluttered background video data, respectively, and outperformed previous methods: Hidden Markov Model (HMM), Finite State Machine (FSM).


Two-layer Bayesian network Hand gesture recognition Virtual mouse interface 



This work was partly supported by the ICT R&D program of MSIP/IITP [B0101-15-0552 , Development of Predictive Visual Intelligence Technology] and also supported by the Implementation of Technologies for Identification, Behavior, and Location of Human based on Sensor Network Fusion Program through the Ministry of Trade, Industry and Energy (Grant No. 10041629).


  1. 1.
    Argyros A, Lourakis M (2006) Vision-based interpretation of hand gestures for remote control of a computer mouse. In: Proc European conference on computer vision, vol 3979. Graz, Austria, pp 40–51Google Scholar
  2. 2.
    Brand J, Mason J (2000) A comparative assessment of three approaches to pixel level human skin-detection 1:1056–1059Google Scholar
  3. 3.
    Bretzner L, Laptev I, Lindeberg T, Lenman S, Sundblad Y (2001) A prototype system for computer vision based human computer interaction, Tech. Rep. CVAP251, Department of Numerical Analysis and Computer Science, KTH (Royal Institute of Technology), Stockholm, SwedenGoogle Scholar
  4. 4.
    Caschera M C, Ferri F, Grifoni P (2013) InteSe: an integrated model for resolving ambiguities in multimodal sentences. IEEE Trans Syst Man Cybern Syst 43 (4):911–931CrossRefGoogle Scholar
  5. 5.
    Dempster A, Laird N, Rubin D (1977) Maximum-likelihood from incomplete data via the EM algorithm. J R Stat Soc 39(1):1–38MathSciNetMATHGoogle Scholar
  6. 6.
    Dorner B, Hagen E (1994) Towards an American sign language interface. Artif Intell Rev 8(2):235–253CrossRefGoogle Scholar
  7. 7.
    El-Sawah A, Georganas N, Petriu E (2008) A prototype for 3-D hand tracking and posture estimation. IEEE Trans Instrument Measur 57(8):1627–1636CrossRefGoogle Scholar
  8. 8.
    Fu Y, Huang T S (2007) hMouse: Head tracking driven virtual computer mouse. In: Proc. of the 8th IEEE workshop on applications of computer vision. Austin, pp 30–36Google Scholar
  9. 9.
    Hu K, Yin L (2015) Multiple feature representations from multi-layer geometric shape for hand gesture analysis. In: Proc. of 11th IEEE international conference on automatic face and gesture recognition. SloveniaGoogle Scholar
  10. 10.
    Isard M, Blake A (1998) Condensation - conditional density propagation for visual tracking. Int J Comput Vis 29:5–28CrossRefGoogle Scholar
  11. 11.
    Isard M, Maccormick J (2001) BraMBLe: a Bayesian multiple-blob tracker 2:34–41Google Scholar
  12. 12.
    Krejov P, Gilbert A, Bowden R (2015) Combining discriminative and model based approaches for hand pose estimation. In: Proc. of 11th IEEE international conference on automatic face and gesture recognition. SloveniaGoogle Scholar
  13. 13.
    Laptev I, Lindeberg T (2001) Tracking of multi-state hand models using particle filtering and a hierarchy of multi-scale image features, pp 63–74Google Scholar
  14. 14.
  15. 15.
    Mitra S, Acharya T (2007) Gesture recognition: a survey. IEEE Trans Syst Man Cybern 37(3):311–324CrossRefGoogle Scholar
  16. 16.
    Murphy K (2002) Dynamic Bayesian networks: representation, inference and learning, Ph.D. thesis, University of British ColumbiaGoogle Scholar
  17. 17.
    Nguyen V-T, Le T-L, Tran T-H, Mullot R, Courboulay V (2015) A new hand representation based on kernels for hand posture recognition. In: Proc. of 11th IEEE international conference on automatic face and gesture recognition. SloveniaGoogle Scholar
  18. 18.
    Raheja JL, Shyam R, Kumar U, Prasad P B (2010) Real-time robotic hand control using hand gestures. Int Conf Mach Learn Comput:12–16Google Scholar
  19. 19.
    Ramamoorthy A, Vaswani N, Chaudhury S, Banerjee S (2003) Recognition of dynamic hand gestures. Pattern Recog 36(9):2069–2081CrossRefMATHGoogle Scholar
  20. 20.
    Robertson P, Laddaga R, Kleek M (2004) Virtual mouse vision based interface, pp 177–193Google Scholar
  21. 21.
    Stenger B, Thayananthan A, Torr P, Cipolla R (2006) Model-based hand tracking using a hierarchical Bayesian filter. IEEE Trans Pattern Anal Mach Intell 28 (9):1372–1384CrossRefMATHGoogle Scholar
  22. 22.
    Suk H-I, Shin B-K, Lee S-W (2010) Hand gesture recognition based on dynamic Bayesian network framework 43(9):3059–3072Google Scholar
  23. 23.
    Vafadar M, Behrad A (2008) Human hand gesture recognition using motion orientation histogram for interaction of handicapped persons with computer. In: Proc. of international conference on image and signal processing, vol 5099. Normandy, France, pp 378–385Google Scholar
  24. 24.
    Viola P, Jones M (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154CrossRefGoogle Scholar
  25. 25.
    Yeasin M, Chaudhuri S (2000) Visual understanding of dynamic hand gestures. Pattern Recog 33(11):1805–1817CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Myung-Cheol Roh
    • 1
  • Dongoh Kang
    • 2
  • Sungju Huh
    • 2
  • Seong-Whan Lee
    • 2
  1. 1.SeoulSouth Korea
  2. 2.Department of Brain and Cognitive EngineeringKorea UniversitySeoulSouth Korea

Personalised recommendations