Advertisement

A System for Large Vocabulary Sign Search

  • Haijing Wang
  • Alexandra Stefan
  • Sajjad Moradi
  • Vassilis Athitsos
  • Carol Neidle
  • Farhad Kamangar
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6553)

Abstract

A method is presented to help users look up the meaning of an unknown sign from American Sign Language (ASL). The user submits a video of the unknown sign as a query, and the system retrieves the most similar signs from a database of sign videos. The user then reviews the retrieved videos to identify the video displaying the sign of interest. Hands are detected in a semi-automatic way: the system performs some hand detection and tracking, and the user has the option to verify and correct the detected hand locations. Features are extracted based on hand motion and hand appearance. Similarity between signs is measured by combining dynamic time warping (DTW) scores, which are based on hand motion, with a simple similarity measure based on hand appearance. In user-independent experiments, with a system vocabulary of 1,113 signs, the correct sign was included in the top 10 matches for 78% of the test queries.

Keywords

American Sign Language recognition gesture recognition 

References

  1. 1.
    Tennant, R.A., Brown, M.G.: The American Sign Language Handshape Dictionary. Gallaudet U. Press, Washington, DCGoogle Scholar
  2. 2.
    Lane, H., Hoffmeister, R.J., Bahan, B.: A Journey into the Deaf-World. DawnSign Press, San Diego (1996)Google Scholar
  3. 3.
    Schein, J.: At home among strangers. Gallaudet U. Press, Washington, DC (1989)Google Scholar
  4. 4.
    Athitsos, V., Neidle, C., Sclaroff, S., Nash, J., Stefan, A., Yuan, Q., Thangali, A.: The American Sign Language lexicon video dataset. In: IEEE Workshop on Computer Vision and Pattern Recognition for Human Communicative Behavior Analysis, CVPR4HB (2008)Google Scholar
  5. 5.
    Kruskal, J.B., Liberman, M.: The symmetric time warping algorithm: From continuous to discrete. In: Time Warps. Addison-Wesley (1983)Google Scholar
  6. 6.
    Bauer, B., Hienz, H., Kraiss, K.F.: Video-based continuous sign language recognition using statistical methods. In: International Conference on Pattern Recognition, pp. 2463–2466 (2000)Google Scholar
  7. 7.
    Dreuw, P., Deselaers, T., Keysers, D., Ney, H.: Modeling image variability in appearance-based gesture recognition. In: ECCV Workshop on Statistical Methods in Multi-Image and Video Processing, pp. 7–18 (2006)Google Scholar
  8. 8.
    Starner, T., Pentland, A.: Real-time American Sign Language recognition using desk and wearable computer based video. IEEE Transactions on Pattern Analysis and Machine Intelligence 20, 1371–1375 (1998)CrossRefGoogle Scholar
  9. 9.
    Vogler, C., Metaxas, D.N.: Parallel hidden markov models for american sign language recognition. In: IEEE International Conference on Computer Vision, ICCV, pp. 116–122 (1999)Google Scholar
  10. 10.
    Cui, Y., Weng, J.: Appearance-based hand sign recognition from intensity image sequences. Computer Vision and Image Understanding 78, 157–176 (2000)CrossRefGoogle Scholar
  11. 11.
    Ke, Y., Sukthankar, R., Hebert, M.: Efficient visual event detection using volumetric features. In: IEEE International Conference on Computer Vision, ICCV, vol. 1, pp. 166–173 (2005)Google Scholar
  12. 12.
    Wang, S.B., Quattoni, A., Morency, L.P., Demirdjian, D., Darrell, T.: Hidden conditional random fields for gesture recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR, vol. 2, pp. 1521–1527 (2006)Google Scholar
  13. 13.
    Kadir, T., Bowden, R., Ong, E., Zisserman, A.: Minimal training, large lexicon, unconstrained sign language recognition. In: British Machine Vision Conference, BMVC, vol. 2, pp. 939–948 (2004)Google Scholar
  14. 14.
    Zieren, J., Kraiss, K.-F.: Robust Person-Independent Visual Sign Language Recognition. In: Marques, J.S., Pérez de la Blanca, N., Pina, P. (eds.) IbPRIA 2005, Part I. LNCS, vol. 3522, pp. 520–528. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  15. 15.
    Valli, C. (ed.): The Gallaudet Dictionary of American Sign Language. Gallaudet U. Press, Washington, DC (2006)Google Scholar
  16. 16.
    Cooper, H., Bowden, R.: Learning signs from subtitles: A weakly supervised approach to sign language recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 2568–2574 (2009)Google Scholar
  17. 17.
    Farhadi, A., Forsyth, D.A., White, R.: Transfer learning in sign language. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2007)Google Scholar
  18. 18.
    Bobick, A., Davis, J.: The recognition of human movement using temporal templates. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI 23, 257–267 (2001)CrossRefGoogle Scholar
  19. 19.
    Gorelick, L., Blank, M., Shechtman, E., Irani, M., Basri, R.: Actions as space-time shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 2247–2253 (2007)CrossRefGoogle Scholar
  20. 20.
    Yao, G., Yao, H., Liu, X., Jiang, F.: Real time large vocabulary continuous sign language recognition based on OP/Viterbi algorithm. In: International Conference on Pattern Recognition, vol. 3, pp. 312–315 (2006)Google Scholar
  21. 21.
    Stefan, A., Wang, H., Athitsos, V.: Towards automated large vocabulary gesture search. In: Conference on Pervasive Technologies Related to Assistive Environments, PETRA (2008)Google Scholar
  22. 22.
    Jones, M., Rehg, J.: Statistical color models with application to skin detection. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. I:274–I:280 (1999)Google Scholar
  23. 23.
    Rowley, H., Baluja, S., Kanade, T.: Rotation invariant neural network-based face detection. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 38–44 (1998)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Haijing Wang
    • 1
  • Alexandra Stefan
    • 1
  • Sajjad Moradi
    • 1
  • Vassilis Athitsos
    • 1
  • Carol Neidle
    • 2
  • Farhad Kamangar
    • 1
  1. 1.Computer Science and Engineering DepartmentUniversity of Texas at ArlingtonArlingtonUSA
  2. 2.Linguistics ProgramBoston UniversityBostonUSA

Personalised recommendations