Sign Language Recognition Using Kinect

  • Simon Lang
  • Marco Block
  • Raúl Rojas
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7267)

Abstract

An open source framework for general gesture recognition is presented and tested with isolated signs of sign language. Other than common systems for sign language recognition, this framework makes use of Kinect, a depth camera which makes real-time 3D-reconstruction easily applicable. Recognition is done using hidden Markov models with a continuous observation density. The framework also offers an easy way of initializing and training new gestures or signs by performing them several times in front of the camera. First results with a recognition rate of 97% show that depth cameras are well-suited for sign language recognition.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Kelly, D., McDonald, J., Markham, C.: Recognizing Spatiotemporal Gestures and Movement Epenthesis in Sign Language. In: 13th International Machine Vision and Image Processing Conference (IMVIP 2009). IEEE Computer Society, Washington, DC (2009)Google Scholar
  2. 2.
    Dreuw, P., Rybach, D., Deselaers, T., Zahedi, M., Ney, H.: Speech Recognition Techniques for a Sign Language Recognition System. In: INTERSPEECH 2007, 8th Annual Conference of the International Speech Communication Association (ISCA 2007), pp. 2513–2516 (2007)Google Scholar
  3. 3.
    Li, X., Parizeau, M., Plamondon, R.: Training Hidden Markov Models with Multiple Observations – A combinatorial Method. IEEE Transactions on PAMI PAMI-22(4), 371–377 (2000)Google Scholar
  4. 4.
    Vogler, C., Metaxas, D.: ASL Recognition Based on a Coupling Between HMMs and 3D Motion Analysis. In: Proceedings of the Sixth International Conference on Computer Vision, pp. 363–369. Narosa Publishing House (1998) ISBN: 978-8-17319-221-0Google Scholar
  5. 5.
    Starner, T., Pentland, A.: Real-Time American Sign Language Recognition from Video Using Hidden Markov Models. In: Proceedings of the International Symposium on Computer Vision, ISCV 1995, pp. 265–270. IEEE Publications, U.S (1995) ISBN: 978-0-81867-190-6CrossRefGoogle Scholar
  6. 6.
    Boyes Braem, P.: Einführung in die Gebärdensprache und ihre Erforschung. In: Internationale Arbeiten zur Gebärdensprache und Kommunikation Gehörloser, 1st edn., vol. 11. SIGNUM-Verlag (1990) ISBN: 978-3-92773-110-3Google Scholar
  7. 7.
    Rabiner, L.R.: A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. Proceedings of the IEEE 77(2), 257–286 (1989)CrossRefGoogle Scholar
  8. 8.
    Rahimi, A.: An Erratum for “A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition”, website of Ali Rahimi at MIT Media Laboratory, http://xenia.media.mit.edu/~rahimi/rabiner/rabiner-errata/rabiner-errata.html
  9. 9.
    Official Microsoft Xbox website, introduction of Kinect, http://www.xbox.com/en-US/kinect
  10. 10.
    Countdown to Kinect: 17 Controller-Free Games Launch in November, Microsoft News Center, https://www.microsoft.com/presspass/press/2010/oct10/10-18mskinectuspr.mspx
  11. 11.
    Kinect Downgraded To Save Money, Can’t Read Sign Language, News at Kotaku, http://kotaku.com/5609840/kinect-dumbed-down-to-save-money-cant-read-sign-language
  12. 12.
    CopyCat and Kinect, overview of the CopyCat Kinect demo on the website of the Center for Accessible Technology in Sign (CATS), http://cats.gatech.edu/content/copycat-and-kinect
  13. 13.
    Integrating Speech and Hearing Challenge Individuals, YouTube channel of Dr. Natheer Khasawneh, http://www.youtube.com/user/knatheer#p/a/u/1/vVL398dUU5Q

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Simon Lang
    • 1
  • Marco Block
    • 1
  • Raúl Rojas
    • 1
  1. 1.Institut für Informatik und MathematikFreie Universität BerlinBerlinGermany

Personalised recommendations