Skip to main content

Boosting real-time recognition of hand posture and gesture for virtual mouse operations with segmentation

Applied Intelligence volume 43, pages 786–801 (2015)Cite this article

Abstract

The design and implementation of polylogarithmically or polynomially bounded algorithms on faster processors has gained popularity and attracted the attention of both researchers and practitioners. The evolution in the computer hardware technology has boosted the development of real-time applications which are expected to respond within a strict time frame. One attractive sophisticated application, which requires real time response, is image capturing and recognition for effective human computer interaction. It is gaining popularity, especially after the development of hand held devices and touch screens. Real-time video processing response time is expressed by means of frame sequences; device dependent capability (20 frame/sec) designates real-time restrictions (a frame is needed to be processed within 50 ms). Video processing of virtual mouse operations requires real-time recognition, i.e., no delay in response can be tolerated. There are indeed several attempts to recognize hand gestures for different purposes. Sign language recognition stands out as the most popular one. However, virtual mouse operations may also be used in general by the majority of people in parallel for the proliferation of different applications on a variety of platforms such as tablet PCs, embedded devices, etc. One significant advantage of such systems fulfills the need for extra hardware system. To this end, we have developed a novel real-time virtual mouse application. Our system architecture recognizes defined postures and gestures. We have implemented, tested, and compared the performance of four methods, namely Chai (static), face (dynamic), regional (dynamic), and Duan. Further, various conditions, such as lighting, distinguishing skin color, and complex background have been considered and discussed.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27

Notes

  1. http://www.cs.brown.edu/~pff/segment (last visited March 13, 2015)

  2. http://www.handresearch.com (last visited March 13, 2015)

  3. http://pages.cpsc.ucalgary.ca/~ozyer/hgr.avi (last visited March 13, 2015)

  4. http://youtu.be/kQxiFaZbOfA (last visited March 13, 2015)

  5. http://ozyer.etu.edu.tr/hgr_source.zip (last visited March 13, 2015)

  6. see footnote 5.

  7. see footnote 5.

  8. see footnote 5.

  9. see footnote 3.

  10. see footnote 4.

References

  1. Cohen I, Sebe N, Garg A, Chen LS, Huang TS (2003) Facial expression recognition from video sequences: temporal and static modeling. Comp Vision Image Underst 91(1):160–187

    Article  Google Scholar 

  2. Yeh C-H, Jiang S-JF, Bai J-C, Liou J-S, Yeh R-N, Wang S-C, Sung P-Y (2010) Vision-based virtual control mechanism via hand gesture recognition. J Comput 21(2)

  3. Ong SCW, Automatic sign language analysis: a survey and the future beyond lexical meaning (2005). IEEE Transactions on Pattern Analysis and Machine Intelligence 27(6):873–891

    Article  Google Scholar 

  4. Rougier C, Meunier J, St-Arnaud A, Rousseau J (2011) Robust video surveillance for fall detection based on human shape deformation. IEEE Transactions on Circuits and Systems for Video Technology 21(5):611–622

    Article  Google Scholar 

  5. Mitra S, Acharya T (2007) Gesture recognition: a survey. IEEE Trans Syst Man Cybern. Part C Appl Rev 37(3):311–324

    Article  Google Scholar 

  6. Aksaç A, Ozturk O, Ozyer T (2011) Real-time multi-objective hand posture/gesture recognition by using distance classifiers and finite state machine for virtual mouse operations. In: Electrical and Electronics Engineering (ELECO), 2011 7th International Conference on, pages II–457. IEEE

  7. Kakumanu P, Makrogiannis S, Bourbakis N (2007) A survey of skin-color modeling and detection methods. Pattern Recognit 40(3):1106–1122

    Article  MATH  Google Scholar 

  8. Fu Y, Huang TS (2007) Head tracking driven virtual computer mouse. In: Applications of computer vision, 2007. WACV’07. IEEE Workshop on. IEEE, pp 30–30

  9. Pallejà T., Soler Edgar R., Teixidó M., Tresanchez M., Fernández del Viso A., Sánchez CR, Palacin J (2008) Using the optical flow to implement a relative virtual mouse controlled by head movements. J UCS 14(19):3127–3141

    Google Scholar 

  10. Xu G, Wang Y, Feng X (2009) A robust low cost virtual mouse based on face tracking. In: Pattern recognition, 2009. Chinese Conference on CCPR 2009, IEEE, pp 1–4

  11. Tsang W-WM, Pun K-P (2005) A finger-tracking virtual mouse realized in an embedded system. In: Intelligent signal processing and communication systems, 2005. Proceedings of 2005 International Symposium on ISPACS 2005. IEEE, pp 781–784

  12. Chai Douglas, Ngan King N (1999) Face segmentation using skin-color map in videophone applications. Circuits and Systems for Video Technology, IEEE Transactions on 9(4):551– 564

    Article  Google Scholar 

  13. Sanghi A, Arora H, Gupta K, Vats VB (2008) A fingertip detection and tracking system as a virtual mouse, a signature input device and an application selector. In: Devices, circuits and systems, 2008. 7th international caribbean conference on ICCDCS 2008, IEEE, pp 1–4

  14. Roh M-C, Huh S-J, Lee S-W (2009) A virtual mouse interface based on two-layered bayesian network. In: 2009 Workshop on Applications of computer vision (WACV), IEEE, pp 1–6

  15. Brand J, Mason JS (2000) A comparative assessment of three approaches to pixel-level human skin-detection. In: Pattern Recognition, 2000. 15th International Conference on Proceedings, vol 1. IEEE, pp 1056–1059

  16. Wang X, Qin K (2010) A six-degree-of-freedom virtual mouse based on hand gestures. In: Proceedings of the 2010 International Conference on Electrical and Control Engineering, IEEE Computer Society, pp 257–260

  17. Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: Computer Vision and Pattern Recognition, 2001. Proceedings of the 2001 IEEE Computer Society Conference on CVPR 2001, vol 1. IEEE, pp I– 511

  18. Vezhnevets V, Sazonov V, Andreeva A (2003) A survey on pixel-based skin color detection techniques. In: Proceedings Graphicon, vol 3. Moscow, pp 85–92

  19. Lienhart R, Maydt J (2002) An extended set of haar-like features for rapid object detection. In: Image processing. 2002. International Conference on Proceedings. 2002 vol 1. IEEE, pp I–900

  20. Bradski GR (1998) Computer vision face tracking for use in a perceptual user interface

  21. Comaniciu D, Meer P (1999) Mean shift analysis and applications. In: The Proceedings of the Seventh IEEE International Conference on Computer vision, 1999 , vol 2. IEEE, pp 1197–1203

  22. Comaniciu D, Meer P (1997) Robust analysis of feature spaces: color image segmentation. In: Computer Vision and Pattern Recognition, 1997. 1997 IEEE Computer Society Conference on Proceedings. IEEE, pp 750–755

  23. Bradski G, Kaehler A (2008) Learning OpenCV: computer vision with the OpenCV library. O’Reilly Media, Inc.

  24. Felzenszwalb PF, Huttenlocher DP (2004) Efficient graph-based image segmentation. Int J Comput Vis 59(2):167–181

    Article  Google Scholar 

  25. Michel D, Oikonomidis I, Argyros A (2011) Scale invariant and deformation tolerant partial shape matching. Image Vis Comput 29(7):459–469

    Article  Google Scholar 

  26. Bayer V (1999) Survey of algorithms for the convex hull problem. preprint

  27. Sklansky J (1982) Finding the convex hull of a simple polygon. Pattern Recogn Lett 1(2):79–83

    Article  MATH  Google Scholar 

  28. Duan L, Cui G, Gao W, Zhang H (2002) Adult image detection method base-on skin color model and support vector machine. In: Asian conference on computer vision, pp 797–800

  29. Zhang H, Fritts JE, Goldman SA (2008) Image segmentation evaluation: a survey of unsupervised methods. Comp Vision Image Underst 110(2):260–280

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Engineering, TOBB University of Economics & Technology, Ankara, Turkey

    Orkun Ozturk, Alper Aksac & Tansel Ozyer

  2. Department of Computer Science, University of Calgary, Calgary, AB, Canada

    Alper Aksac & Reda Alhajj

Authors
  1. Orkun Ozturk
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alper Aksac
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tansel Ozyer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Reda Alhajj
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tansel Ozyer.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Ozturk, O., Aksac, A., Ozyer, T. et al. Boosting real-time recognition of hand posture and gesture for virtual mouse operations with segmentation. Appl Intell 43, 786–801 (2015). https://doi.org/10.1007/s10489-015-0680-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-015-0680-z

Keywords

  • Real-time analysis
  • Hand segmentation
  • Hand posture/gesture recognition
  • Human computer interaction
  • Skin color detection
  • Virtual mouse operations