Vision-Based Portuguese Sign Language Recognition System

  • Paulo Trigueiros
  • Fernando Ribeiro
  • Luís Paulo Reis
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 275)

Abstract

Vision-based hand gesture recognition is an area of active current research in computer vision and machine learning. Being a natural way of human interaction, it is an area where many researchers are working on, with the goal of making human computer interaction (HCI) easier and natural, without the need for any extra devices. So, the primary goal of gesture recognition research is to create systems, which can identify specific human gestures and use them, for example, to convey information. For that, vision-based hand gesture interfaces require fast and extremely robust hand detection, and gesture recognition in real time. Hand gestures are a powerful human communication modality with lots of potential applications and in this context we have sign language recognition, the communication method of deaf people. Sign languages are not standard and universal and the grammars differ from country to country. In this paper, a real-time system able to interpret the Portuguese Sign Language is presented and described. Experiments showed that the system was able to reliably recognize the vowels in real-time, with an accuracy of 99.4% with one dataset of features and an accuracy of 99.6% with a second dataset of features. Although the implemented solution was only trained to recognize the vowels, it is easily extended to recognize the rest of the alphabet, being a solid foundation for the development of any vision-based sign language recognition user interface system.

Keywords

Sign Language Recognition Hand Gestures Hand Postures Gesture Classification Computer Vision Machine Learning 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Wikipedia. Língua gestual portuguesa. 2012 September 9 (2013), http://pt.wikipedia.org/wiki/Lingua_gestual_portuguesa (cited 2013)
  2. 2.
    Vijay, P.K., et al.: Recent Developments in Sign Language Recognition: A Review. International Journal on Advanced Computer Engineering and Communication Technology 1(2), 21–26 (2012)Google Scholar
  3. 3.
    Mingqiang, Y., Idiyo, K., Joseph, R.: A Survey of Shape Feature Extraction Techniques. Pattern Recognition, 43–90 (2008)Google Scholar
  4. 4.
    Miner, R.: RapidMiner: Report the Future (December 2011), http://rapid-i.com/
  5. 5.
    Mitra, S., Acharya, T.: Gesture recognition: A Survey. IEEE Transactions on Systems, Man and Cybernetics, 311–324 (2007)Google Scholar
  6. 6.
    Murthy, G.R.S., Jadon, R.S.: A Review of Vision Based Hand Gestures Recognition. International Journal of Information Technology and Knowledge Management 2(2), 405–410 (2009)Google Scholar
  7. 7.
    Wachs, J.P., Stern, H., Edan, Y.: Cluster Labeling and Parameter Estimation for the Automated Setup of a Hand-Gesture Recognition System. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans 35(6), 932–944 (2005)CrossRefGoogle Scholar
  8. 8.
    Trigueiros, P., Ribeiro, F., Reis, L.P.: A Comparative Study of different image features for hand gesture machine learning. In: 5th International Conference on Agents and Artificial Intelligence, Barcelona, Spain (2013)Google Scholar
  9. 9.
    Trigueiros, P., Ribeiro, F.: Vision-based Hand WheelChair Control. In: 12th International Conference on Autonomous Robot Systems and Competitions, Guimarães, Portugal (2012)Google Scholar
  10. 10.
    Wang, C.-C., Wang, K.-C.: Hand Posture Recognition Using Adaboost with SIFT for Human Robot Interaction. In: Proceedings of the International Conference on Advanced Robotics 2008, Jeju, Korea (2008)Google Scholar
  11. 11.
    Conseil, S., Bourenname, S., Martin, L.: Comparison of Fourier Descriptors and Hu Moments for Hand Posture Recognition. In: 15th European Signal Processing Conference (EUSIPCO) 2007, Poznan, Poland, pp. 1960–1964 (2007)Google Scholar
  12. 12.
    Barczak, A.L.C., et al.: Analysis of Feature Invariance and Discrimination for Hand Images: Fourier Descriptors versus Moment Invariants. In: International Conference Image and Vision Computing 2011, New Zeland (2011)Google Scholar
  13. 13.
    Bourennane, S., Fossati, C.: Comparison of shape descriptors for hand posture recognition in video. Signal, Image and Video Processing 6(1), 147–157 (2010)CrossRefGoogle Scholar
  14. 14.
    Triesch, J., Malsburg, C.V.D.: Robust Classification of Hand Postures against Complex Backgrounds. In: International Conference on Automatic Face and Gesture Recognition, Killington, Vermont, USA (1996)Google Scholar
  15. 15.
    Huynh, D.Q.: Evaluation of Three Local Descriptors on Low Resolution Images for Robot Navigation. In: 24th International Conference Image and Vision Computing, Wellington, New Zealand (2009)Google Scholar
  16. 16.
    Fang, Y., et al.: Hand Posture Recognition with Co-Training. In: 19th International Conference on Pattern Recognition, Tampa, FL, USA (2008)Google Scholar
  17. 17.
    Blum, A., Mitchell, T.: Combining labeled and unlabeled data with co-training. In: Proceedings of the Eleventh Annual Conference on Computational Learning Theory 1998, pp. 92–100. ACM, Madison (1998)Google Scholar
  18. 18.
    Tara, R.Y., Santosa, P.I., Adji, T.B.: Sign Language Recognition in Robot Teleoperation using Centroid Distance Fourier Descriptors. International Journal of Computer Applications 48(2) (2012)Google Scholar
  19. 19.
    Faria, B.M., Lau, N., Reis, L.P.: Classification of Facial Expressions Using Data Mining and machine Learning Algorithms. In: 4a Conferência Ibérica de Sistemas e Tecnologias de Informação, Póvoa de Varim, Portugal (2009)Google Scholar
  20. 20.
    Faria, B.M., Reis, L.P., Lau, N.: Cerebral Palsy EEG Signals Classification: Facial Expressions and Thoughts for Driving an Intelligent Wheelchair. In: 2012 IEEE 12th International Conference on Data Mining Workshops (ICDMW) (2012)Google Scholar
  21. 21.
    Gillian, N.E.: Gesture Recognition for Musician Computer Interaction. In: Music Department 2011, Faculty of Arts, p. 206. Humanities and Social Sciences, Belfast (2011)Google Scholar
  22. 22.
    Faria, B.M., et al.: Machine Learning Algorithms applied to the Classification of Robotic Soccer Formations and Opponent Teams. In: IEEE Conference on Cybernetics and Intelligent Systems (CIS) 2010, Singapore, pp. 344–349 (2010)Google Scholar
  23. 23.
    Mannini, A., Sabatini, A.M.: Machine learning methods for classifying human physical activity from on-body accelerometers. Sensors (Basel) 10(2), 1154–1175 (2010)CrossRefGoogle Scholar
  24. 24.
    Maldonado-Báscon, S., et al.: Road-Sign detection and Recognition Based on Support Vector Machines. IEEE Transactions on Intelligent Transportation Systems, 264–278 (2007)Google Scholar
  25. 25.
    Vicen-Bueno, R., et al.: Complexity Reduction in Neural Networks Appplied to Traffic Sign Recognition Tasks (2004)Google Scholar
  26. 26.
    Trigueiros, P., Ribeiro, F., Reis, L.P.: A comparison of machine learning algorithms applied to hand gesture recognition. In: 7th Iberian Conference on Information Systems and Technologies, Madrid, Spain (2012)Google Scholar
  27. 27.
    Rego, P., Moreira, P.M., Reis, L.P.: Serious games for rehabilitation: A survey and a classification towards a taxonomy. In: 2010 5th Iberian Conference on Information Systems and Technologies (CISTI) (2010)Google Scholar
  28. 28.
    Rego, P.A., Moreira, P.M., Reis, L.P.: Natural user interfaces in serious games for rehabilitation. In: 2011 6th Iberian Conference on Information Systems and Technologies, CISTI (2011)Google Scholar
  29. 29.
    Mendes, L., et al.: Virtual centre for the rehabilitation of road accident victims (VICERAVI). In: 2012 7th Iberian Conference on Information Systems and Technologies, CISTI (2012)Google Scholar
  30. 30.
    Rego, P.A., Moreira, P.M., Reis, L.P.: New Forms of Interaction in Serious Games for Rehabilitation. In: Handbook of Research on Serious Games as Educational, Business and Research Tools, pp. 1188–1211. IGI Global (2012)Google Scholar
  31. 31.
    Montesano, L., et al.: Towards an Intelligent Wheelchair System for Users With Cerebral Palsy. IEEE Transactions on Neural Systems and Rehabilitation Engineering 18(2), 193–202 (2010)CrossRefGoogle Scholar
  32. 32.
    Braga, R.A., et al.: IntellWheels: modular development platform for intelligent wheelchairs. J. Rehabil. Res. Dev. 48(9), 1061–1076 (2011)CrossRefGoogle Scholar
  33. 33.
    Faria, B.M., et al.: Evaluation of distinct input methods of an intelligent wheelchair in simulated and real environments: a performance and usability study. Assistive Technology: The Official Journal of RESNA 25(2), 88–98 (2013)CrossRefMathSciNetGoogle Scholar
  34. 34.
    Faria, B.M., et al.: A Methodology for Creating Intelligent Wheelchair Users’ Profiles. In: 4th International Conference on Agents and Artificial Intelligence, Algarve, Portugal (2012)Google Scholar
  35. 35.
    Ke, W., et al.: Real-Time Hand Gesture Recognition for Service Robot, pp. 976–979 (2010)Google Scholar
  36. 36.
    Oshita, M., Matsunaga, T.: Automatic learning of gesture recognition model using SOM and SVM. In: Bebis, G., et al. (eds.) ISVC 2010, Part I. LNCS, vol. 6453, pp. 751–759. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  37. 37.
    Almeida, R., Reis, L.P., Jorge, A.M.: Analysis and Forecast of Team Formation in the Simulated Robotic Soccer Domain. In: Lopes, L.S., Lau, N., Mariano, P., Rocha, L.M. (eds.) EPIA 2009. LNCS, vol. 5816, pp. 239–250. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  38. 38.
    Oka, K., Sato, Y., Koike, H.: Real-time fingertip tracking and gesture recognition. IEEE Computer Graphics and Applications 22(6), 64–71 (2002)CrossRefGoogle Scholar
  39. 39.
    Perrin, S., Cassinelli, A., Ishikawa, M.: Gesture recognition using laser-based tracking system. In: Sixth IEEE International Conference on Automatic Face and Gesture Recognition, Seoul, South Korea (2004)Google Scholar
  40. 40.
    Binh, N.D., Shuichi, E., Ejima, T.: Real-Time Hand Tracking and Gesture Recognition System. In: Proceedings of International Conference on Graphics, Vision and Image 2005, Cairo, Egypt (2005)Google Scholar
  41. 41.
    Chen, F.-S., Fu, C.-M., Huang, C.-L.: Hand gesture recognition using a real-time tracking method and hidden Markov models. Image and Vision Computing 21(8), 745–758 (2003)CrossRefGoogle Scholar
  42. 42.
    Kelly, D., McDonald, J., Markham, C.: Recognition of Spatiotemporal Gestures in Sign Language Using Gesture Threshold HMMs. In: Wang, L., et al. (eds.) Machine Learning for Vision-Based Motion Analysis, pp. 307–348. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  43. 43.
    Zafrulla, Z., et al.: American sign language recognition with the kinect. In: 13th International Conference on Multimodal Interfaces 2011, pp. 279–286. ACM, Alicante (2011)Google Scholar
  44. 44.
    Chowdhury, J.R.: Kinect Sensor for Xbox Gaming, IIT Kharagpur (2012)Google Scholar
  45. 45.
    Andersen, M.R., et al.: Kinect Depth Sensor Evaluation for Computer Vision Applications. In: Technical report ECE-TR-6 2012, Department of Engineering – Electrical and Computer Engineering, Aarhus University, p. 37 (2012)Google Scholar
  46. 46.
    Cooper, H., Bowden, R.: Large lexicon detection of sign language. In: Lew, M., Sebe, N., Huang, T.S., Bakker, E.M. (eds.) HCI 2007. LNCS, vol. 4796, pp. 88–97. Springer, Heidelberg (2007)Google Scholar
  47. 47.
    Trigueiros, P., Ribeiro, F., Reis, L.P.: Vision Based Referee Sign Language Recognition System for the RoboCup MSL League. In: 17th Annual Robocup International Symposium, Eindhoven, Holland (2013)Google Scholar
  48. 48.
    Trigueiros, P., Ribeiro, F., Reis, L.P.: Vision-based Gesture Recognition System for Human-Computer Interaction. In: IV ECCOMAS Thematic Conference on Computational Vision and Medical Image Processing. Taylor and Francis, Publication, Funchal (2013)Google Scholar
  49. 49.
    Lieberman, Z., Watson, T., Castro, A.: openFrameworks. 2004 10 October (2013) [cited 2011; openFrameworks is an open source C++ toolkit designed to assist the creative process by providing a simple and intuitive framework for experimentation], http://www.openframeworks.cc/
  50. 50.
    Bradski, G., Kaehler, A.: Learning OpenCV: Computer Vision with the OpenCV Library, 1st edn. O’Reilly Media (2008)Google Scholar
  51. 51.
    OpenNI. The standard framework for 3D sensing (2013), http://www.openni.org/
  52. 52.
    King, D.E.: Dlib-ml: A Machine Learning Toolkit. Journal of Machine Learning Research 10, 1755–1758 (2009)Google Scholar
  53. 53.
    Zhang, D., Lu, G.: A comparative Study of Fourier Descriptors for Shape Representation and Retrieval. In: Proc. of 5th Asian Conference on Computer Vision (ACCV). Springer, Melbourne (2002)Google Scholar
  54. 54.
    Kauppinen, H., Seppanen, T., Pietikainen, M.: An experimental comparison of autoregressive and Fourier-based descriptors in 2D shape classification. IEEE Transactions on Pattern Analysis and Machine Intelligence 17(2), 201–207 (1995)CrossRefGoogle Scholar
  55. 55.
    Zhang, D., Lu, G.: A Comparative Study on Shape Retrieval Using Fourier Descriptors with Different Shape Signatures. Journal of Visual Communication and Image Representation 14(1), 41–60 (2003)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Paulo Trigueiros
    • 1
  • Fernando Ribeiro
    • 2
  • Luís Paulo Reis
    • 3
  1. 1.Instituto Politécnico do PortoPortoPortugal
  2. 2.Departamento de Electrónica Industrial da Universidade do MinhoGuimarãesPortugal
  3. 3.EEUM – Escola de Engenharia da Universidade do Minho – DSIGuimarãesPortugal

Personalised recommendations