Skip to main content
SpringerLink
Log in

Generic System for Human-Computer Gesture Interaction: Applications on Sign Language Recognition and Robotic Soccer Refereeing

  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

Hand gestures are a powerful way for human communication, with lots of potential applications in the area of human computer interaction. Vision-based hand gesture recognition techniques have many advantages compared with traditional devices, giving users a simpler and more natural way to communicate with electronic devices. This work proposes a generic system architecture based in computer vision and machine learning, able to be used with any interface for real-time human-machine interaction. Its novelty is the integration of different tools for gesture spotting and the proposed solution is mainly composed of three modules: a pre-processing and hand segmentation module, a static gesture interface module and a dynamic gesture interface module. The experiments showed that the core of vision-based interaction systems could be the same for all applications and thus facilitate the implementation. For hand posture recognition, a SVM (Support Vector Machine) model was trained with a centroid distance dataset composed of 2170 records, able to achieve a final accuracy of 99.4 %. For dynamic gestures, an HMM (Hidden Markov Model) model was trained for each one of the defined gestures that the system should recognize with a final average accuracy of 93.7 %. The datasets were built from four different users with a total of 25 gestures per user, totalling 1100 records for model construction. The proposed solution has the advantage of being generic enough with the trained models able to work in real-time, allowing its application in a wide range of human-machine applications. To validate the proposed framework two applications were implemented. The first one is a real-time system able to interpret the Portuguese Sign Language. The second one is an online system able to help a robotic soccer game referee judge a game in real-time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Mitra, S., Acharya, T.: Gesture recognition: A Survey. In: IEEE Transactions on Systems, Man and Cybernetics, pp. 311–324. IEEE (2007)

  2. Bourennane, S., Fossati, C.: Comparison of shape descriptors for hand posture recognition in video. Sig. Image Video Process. 6(1), 147–157 (2010)

    Article  Google Scholar 

  3. Yoon, J.-H., Park, J.-S., Sung, M.Y.: Vision-based barehand gesture interface for interactive augmented reality applications. In: 5th International Conference on Entertainment Computing. Springer-Verlag, Cambridge (2006)

  4. Buchmann, V., et al.: FingARtips: Gesture based direct manipulation in Augmented Reality. In: 2nd International Conference on Computer Graphics and Interactive Techniques in Australasia and South East Asia, pp. 212–221. ACM, Singapore (2004)

  5. Trigueiros, P., Ribeiro, F., Lopes, G.: Vision-based hand segmentation techniques for human-robot interaction for real-time applications. In: III ECCOMAS Thematic Conference on Computational Vision and Medical Image Processing. Taylor and Francis, Publication, Olhão, Algarve, Portugal (2011)

  6. Vatavu, R.-D., Anthony, L., Wobbrock, J.O.: Gestures as point clouds: a $P recognizer for user interface prototypes. In: 14th ACM International Conference on Multimodal Interaction, pp. 273–280. ACM, California (2012)

  7. Wobbrock, J.O., Wilson, A.D., Li, Y.: Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes. In: Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology, pp. 159–168. ACM, Newport, Rhode Island (2007)

  8. Li, Y.: Protractor: A fast and accurate gesture recognizer. In: Conference on Human Factors in Computing Systems, pp. 2169–2172. ACM, Atlanta, Georgia (2010)

  9. Kratz, S., Rohs, M.: Protractor3D: A closed-form solution to rotation-invariant 3D gestures. In: 16th International Conference on Intelligent User Interfaces, pp. 371–374. ACM, Palo Alto (2011)

  10. Kim, T.: In-Depth: Eye To Eye - The History Of EyeToy. 2008 November 6, 2008 [cited 2013 29-03-2013]; Available from: http://www.gamasutra.com/php-bin/news_index.php?story=20975

  11. Chowdhury, J.R.: Kinect Sensor for Xbox Gaming. IIT Kharagpur (2012)

  12. Zafrulla, Z. et al.: American sign language recognition with the kinect. In: 13th International Conference on Multimodal Interfaces, pp. 279–286. ACM, Alicante, Spain (2011)

  13. Ong, S.C., Ranganath, S.: Automatic sign language analysis: A survey and the future beyond lexical meaning. IEEE Trans. Pattern Anal. Mach. Intell. 27(6), 873–91 (2005)

  14. Holt, G.A.t. et al.: Influence of handshape information on automatic sign language recognition. In: 8th International Conference on Gesture in Embodied Communication and Human-Computer Interaction. Springer-Verlag, Bielefeld, Germany (2010)

  15. Tara, R.Y., Santosa, P.I., Adji, T.B.: Sign language recognition in robot teleoperation using centroid distance fourier descriptors. Int. J. Comput. Appl. 48(2) (2012)

  16. Trigueiros, P., Ribeiro, F., Reis, L.P.: A Comparative Study of different image features for hand gesture machine learning. In: 5th International Conference on Agents and Artificial Intelligence. Barcelona, Spain (2013)

  17. Murthy, G.R.S., Jadon, R.S.: A review of vision based hand gestures recognition. Int. J. Inf. Technol. Knowl. Manag. 2(2), 405–410 (2009)

  18. Maung, T.H.H.: Real-time hand tracking and gesture recognition system using neural networks. Proc. World Acad. Sci. Eng. Technol. 50, 466–470 (2009)

  19. Snyder, W.E., Qi, H.: Machine Vision. Cambridge University Press (2004)

  20. Vicen-Bueno, R., et al.: Complexity Reduction. In: Neural Networks Appplied to Traffic Sign Recognition Tasks (2004)

  21. Bailador, G., et al.: Real time gesture recognition using continuous time recurrent neural networks. In: 2nd International Conference on Body Area Networks. ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), Florence, Italy (2007)

  22. Ben-Hur, A., Weston, J.: A User’s Guide to Support Vector Machines. In: Data Mining Techniques for the Life Sciences, pp. 223–239. Humana Press (2008)

  23. Faria, B.M., Lau, N., Reis, L.P.: Classification of Facial Expressions Using Data Mining and machine Learning Algorithms. In: 4a Conferência Ibérica de Sistemas e Tecnologias de Informação. Póvoa de Varim, Portugal (2009)

  24. Faria, B.M., et al.: Machine Learning Algorithms applied to the Classification of Robotic Soccer Formations and Opponent Teams. In: IEEE Conference on Cybernetics and Intelligent Systems (CIS), pp. 344–349. Singapore (2010)

  25. Ke, W., et al.: Real-Time Hand Gesture Recognition for Service Robot, pp. 976–979 (2010)

  26. Maldonado-Báscon, S., et al.: Road-Sign detection and Recognition Based on Support Vector Machines. In: IEEE Transactions on Intelligent Transportation Systems, pp. 264–278 (2007)

  27. Oshita, M., Matsunaga, T.: Automatic learning of gesture recognition model using SOM and SVM. In: International Conference on Advances in Visual Computing. Springer-Verlag, Las Vegas (2010)

  28. Almeida, R., Reis, L.P., Jorge, A.M.: Analysis and Forecast of Team Formation in the Simulated Robotic Soccer Domain. In: Proceedings of the 14th Portuguese Conference on Artificial Intelligence: Progress in Artificial Intelligence, pp. 239–250. Springer-Verlag, Aveiro, Portugal (2009)

  29. Trigueiros, P., Ribeiro, F., Reis, L.P.: A comparison of machine learning algorithms applied to hand gesture recognition. In: 7th Iberian Conference on Information Systems and Technologies. Madrid, Spain (2012)

  30. Trigueiros, P., Ribeiro, F., Reis, L.P.: Vision-based Sign Language Recognition System. In: World Conference on Information Systems and Technologies. Madeira, Portugal (2014)

  31. Trigueiros, P., Ribeiro, F., Reis, L.P.: Vision-based Gesture Recognition System for Human-Computer Interaction. In: IV ECCOMAS Thematic Conference on Computational Vision and Medical Image Processing. Taylor and Francis, Publication, Funchal, Madeira (2013)

  32. Rabiner, L.R.: A tutorial on hidden Markov models and selected applications in speech recognition. IEEE Proc. 77(2), 257–286 (1989)

    Article  Google Scholar 

  33. Oka, K., Sato, Y., Koike, H.: Real-time fingertip tracking and gesture recognition. IEEE Comput. Graph. Appl. 22(6), 64–71 (2002)

    Article  Google Scholar 

  34. Perrin, S., Cassinelli, A., Ishikawa, M.: Gesture recognition using laser-based tracking system. In: Sixth IEEE International Conference on Automatic Face and Gesture Recognition. Seoul, South Korea (2004)

  35. Binh, N.D., Shuichi, E., Ejima, T.: Real-Time Hand Tracking and Gesture Recognition System. In: Proceedings of International Conference on Graphics, Vision and Image. Cairo - Egypt (2005)

  36. Chen, F.-S., Fu, C.-M., Huang, C.-L.: Hand gesture recognition using a real-time tracking method and hidden Markov models. Image Vis. Comput. 21(8), 745–758 (2003)

    Article  Google Scholar 

  37. Kelly, D., McDonald, J., Markham, C. In: Wang, L., et al. (eds.): Recognition of Spatiotemporal Gestures in Sign Language Using Gesture Threshold HMMs. In: Machine Learning for Vision-Based Motion Analysis, pp. 307–348. Springer, London (2011)

  38. Trigueiros, P., Ribeiro, F., Reis, L.P.: Vision Based Referee Sign Language Recognition System for the RoboCup MSL League. In: 17th annual RoboCup International Symposium. Eindhoven, Holland (2013)

  39. Helen, C., Richard, B.: Large lexicon detection of sign language. In: 11th International Conference on Human-Computer Interaction. Springer-Verlag, Rio de Janeiro, Brazil (2007)

  40. Milosevic, B., Farella, E., Benini, L.: Continuous Gesture Recognition for Resource Constrained Smart Objects. In: The Fourth International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies. Florence, Italy (2010)

  41. Elmezain, M., et al.: A Hidden Markov Model-based Continuous Gesture Recognition System for Hand Motion Trajectory. In: 19th International Conference on Pattern Recognition. Tampa, Florida (2008)

  42. Wachs, J.P., Stern, H., Edan, Y.: Cluster labeling and parameter estimation for the automated setup of a hand-gesture recognition system. IEEE Trans. Syst. Man Cybern. - Part A Syst. Humans 35(6), 932–944 (2005)

  43. Jain, A.K., Duin, R.P.W., Mao, J.: Statisitical pattern recognition: A review. IEEE Trans. Pattern Anal. Mach. Intell. 22(1), 4–36 (2000)

  44. Zhang, D., Lu, G.: A comparative Study of Fourier Descriptors for Shape Representation and Retrieval. In: Proceedings of 5th Asian Conference on Computer Vision (ACCV). Springer, Melbourne, Australia (2002)

  45. Kauppinen, H., Seppanen, T., Pietikainen, M.: An experimental comparison of autoregressive and Fourier-based descriptors in 2D shape classification. IEEE Trans. Pattern Anal. Mach. Intell. 17(2), 201–207 (1995)

    Article  Google Scholar 

  46. Zhang, D., Lu, G.: A comparative study on shape retrieval using fourier descriptors with different shape signatures. J. Vis. Commun. Image Represent. 14(1), 41–60 (2003)

    Article  Google Scholar 

  47. Alpaydin, E.: Introduction to Machine Learning. MIT Press (2004)

  48. Montgomery, D.C., Runger, G.C.: Applied Statistics and Probability for Engineers. Wiley (1994)

  49. Witten, I.H., Frank, E., Hall, M.A. Data Mining - Pratical Machine Learning Tools and Techniques, 3rd edn. Elsevier (2011)

  50. Theodoridis, S., Koutroumbas, K.: Pattern Recognition, 4th edn. Elsevier (2009)

  51. Sayad, D.S.: Support Vector Machine - Classification (SVM). 2010 8 Nov 2012]; Available from: http://www.saedsayad.com/support_vector_machine.htm

  52. Theodoridis, S., Koutroumbas, K.: An Introduction to Pattern Recognition: A Matlab Approach. Academic Press (2010)

  53. Rabiner, L.R., Juang, B.H.: An introduction to hidden Markov models. IEEE ASSp Magazine (1986)

  54. Wu, Y., Huang, T.S.: Vision-Based Gesture Recognition: A Review. In: Proceedings of the International Gesture Workshop on Gesture-Based Communication in Human-Computer Interaction, pp. 103–115. Springer-Verlag (1999)

  55. Fink, G.A.: Markov Models for Pattern recognition - From Theory to Applications, p 248. Springer (2008)

  56. Camastra, F., Vinciarelli, A.: Machine Learning for Audio, Image and Video Analysis. Springer (2008)

  57. Reis, L.P., Lau, N.: COACH UNILANG - A Standard Language for Coaching a (Robo) Soccer Team. In: Birk, V.A., Coradeschi, S., Tadokoro, S. (eds.) RoboCup 2001: Robot Soccer World Cup, pp. 183–192. Springer Berlin Heidelberg (2002)

  58. Backus, J.W., et al.: Revised Report on the Algorithmic Language ALGOL 60. In: Communications of the ACM, pp. 1–17. ACM (1960)

  59. Buckland, M.: Programming Game AI by Example, p 521. Wordware Publishing, Inc. (2005)

  60. Millington, I., Funge, J.: In: Kaufmann, M. (ed.) Artificial Intelligence for Games, 2nd edn, p. 870. Elsevier (2009)

  61. Lieberman, Z., Watson, T., Castro, A.: openFrameworks. 2004 10 October 2013 [cited 2011; openFrameworks is an open source C ++ toolkit designed to assist the creative process by providing a simple and intuitive framework for experimentation]. Available from: http://www.openframeworks.cc/

  62. Bradski, G., Kaehler, A.: Learning OpenCV: Computer Vision with the OpenCV Library, 1st edn. OReilly Media (2008)

  63. OpenNI. The standard framework for 3D sensing. 2013; Available from: http://www.openni.org/

  64. King, D.E.: Dlib-ml: A machine learning toolkit. J. Mach. Learn. Res. 10, 1755–1758 (2009)

    Google Scholar 

  65. Murphy, K.: Hidden Markov Model (HMM) Toolbox for Matlab. 1998 8 June 2005 [cited 2012; This toolbox supports inference and learning for HMMs with discrete outputs (dhmm’s), Gaussian outputs (ghmm’s), or mixtures of Gaussians output (mhmm’s). The Gaussians can be full, diagonal, or spherical (isotropic). It also supports discrete inputs, as in a POMDP. The inference routines support filtering, smoothing, and fixed-lag smoothing.]. Available from: http://www.cs.ubc.ca/_murphyk/Software/HMM/hmm.html

  66. Miner, R.: RapidMiner : Report the Future. December 2011]; Available from: http://rapid-i.com/

  67. Ching-Tang, H., et al.: A real time hand gesture recognition system based on DFT and SVM. In: 8th International Conference on Information Science and Digital Content Technology. Jeju, Korea (2012)

  68. Yen-Ting, C., Kuo-Tsung, T.: Multiple-angle Hand Gesture Recognition by Fusing SVM Classifiers. In: IEEE International Conference on Automation Science and Engineering. Washington, DC (2007)

  69. Gao, J., Cao, Q.: Adaptive HOG-LBP Based Learning for Palm Tracking. In: 2nd International Conference on Computer and Information Applications. Taiyuan, Shanxi, China (2012)

  70. Liu, Y., Zhang, P.: An Automatic Hand Gesture Recognition System Based on Viola-Jones Method and SVMs. In: Second International Workshop on Computer Science and Engineering, WCSE ’09 (2009)

  71. Dardas, N.H., Georganas, N.D.: Real-time hand gesture detection and recognition using bag-of-features and support vector machine techniques. IEEE Trans. Instrum. Meas. 60(11), 3592–3607 (2011)

    Article  Google Scholar 

  72. Yoon, H.-S., et al.: Hand gesture recognition using combined features of location, angle and velocity. Pattern Recogn. 34(7), 1491–1501 (2001)

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Escola de Engenharia da Universidade do Minho, Department Eletrónica Industrial, Guimarães, Portugal

    Paulo Trigueiros & Fernando Ribeiro

  2. Departamento de Informática, Instituto Politécnico do Porto, Guimarães, Portugal

    Paulo Trigueiros

  3. Department Sistemas de Informação, Escola de Engenharia da Universidade do Minho, Guimãraes, Portugal

    Luis Paulo Reis

  4. Centro ALGORITMI, Universidade do Minho, Guimarães, Portugal

    Paulo Trigueiros, Fernando Ribeiro & Luis Paulo Reis

  5. Laboratório de Inteligência Artificial e Ciência de Computadores, Porto, Portugal

    Paulo Trigueiros & Luis Paulo Reis

Authors
  1. Paulo Trigueiros
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fernando Ribeiro
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Luis Paulo Reis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Paulo Trigueiros.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Trigueiros, P., Ribeiro, F. & Reis, L.P. Generic System for Human-Computer Gesture Interaction: Applications on Sign Language Recognition and Robotic Soccer Refereeing. J Intell Robot Syst 80, 573–594 (2015). https://doi.org/10.1007/s10846-015-0192-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10846-015-0192-4

Keywords

Search

Navigation