A Natural Interface for Sign Language Mathematics

  • Nicoletta Adamo-Villani
  • Bedřich Beneš
  • Matt Brisbin
  • Bryce Hyland
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4291)


The general goal of our research is the creation of a natural and intuitive interface for input and recognition of American Sign Language (ASL) math signs. The specific objective of this work is the development of two new interfaces for the Mathsignertm application. Mathsignertm is an interactive, 3D animation-based game designed to increase the mathematical skills of deaf children. The program makes use of standard input devices such as mouse and keyboard. In this paper we show a significant extension of the application by proposing two new user interfaces: (1) a glove-based interface, and (2) an interface based on the use of a specialized keyboard. So far, the interfaces allow for real-time input and recognition of the ASL numbers zero to twenty.


Sign Language Gesture Recognition Hand Gesture American Sign Language Deaf Child 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Holt, J.A., Traxler, C., Allen, T.: Interpreting the scores: A user’s guide to the 9th edition stanford achievement test for educators of deaf and hard-of-hearing students. Technical report, Gallaudet Research Institute, Washington, DC (1997)Google Scholar
  2. 2.
    Caccamise, F., Lang, H.: Signs for Science and Mathematics: A Resource Book for Teachers and Students. Rochester, NY, National Technical Institute for the Deaf, RIT (1996)Google Scholar
  3. 3.
    Adamo-Villani, N., Doublestein, J., Martin, Z.: The mathsigner: An interactive learning tool for american sign language k-3 mathematics. In: IEEE Proceedings of IV04 - 8th International Conference on Information Visualization, pp. 713–716. IEEE Computer Society Press, Los Alamitos (2004)Google Scholar
  4. 4.
    Corporation, I.: 3d interaction (2001),
  5. 5.
    Adamo-Villani, N., Beni, G.: Keyboard encoding of hand gestures. In: Proceedings of HCI International - 10th International Conference on Human-Computer Interaction, vol. 2, pp. 571–575 (2003)Google Scholar
  6. 6.
    Adamo-Villani, N., Beni, G.: Reconfigurable keyboard for gesture control. In: HCI International-11th International Conference on Human-Computer Interaction, on a CD-ROM (2005)Google Scholar
  7. 7.
    Frishberg, N., Corazzo, S., Day, L., Wilcox, S., Schulmeister, R.: Sign language interfaces. In: Proc. of INTERCHI 1993, Amsterdam, The Netherlands, pp. 194–197 (1993)Google Scholar
  8. 8.
    Huang, T., Pavlovic, V.: Hand gesture modeling, analysis, and synthesis. In: Proceedings of the International Workshop on Automatic Face and Gesture Recognition, Zurich (1995)Google Scholar
  9. 9.
    Geer, D.: Will gesture-recognition technology point the way? Computer 37, 20–23 (2004)Google Scholar
  10. 10.
    Yi, B., Harris Jr., F.C., Wang, L., Yan, Y.: Real-time natural hand gestures. Computing in Science and Engineering 7, 92–96 (2005)Google Scholar
  11. 11.
    Sturman, D.J.: Whole-Hand Input. PhD thesis, Massachusetts Institute of Technology, Cambridge, MA, USA (1992)Google Scholar
  12. 12.
    Sturman, D.J., Zeltzer, D.: A survey of glove-based input. IEEE Comput. Graph. Appl. 14, 30–39 (1994)CrossRefGoogle Scholar
  13. 13.
    Culver, V.R.: A hybrid sign language recognition system. In: IEEE Proceedings of the 8th International Symposium on Wearable Computers ISWC 2004, vol. 00, pp. 30–33. IEEE Computer Society, Los Alamitos (2004)Google Scholar
  14. 14.
    Vamplew, P.: Recognition of sign language gestures using neural networks. In: Proc. of 1st Euro. Conf. Disability, Virtual Reality Assoc. Tech., Maidenhead, UK, pp. 27–33 (1996)Google Scholar
  15. 15.
    Starner, T., Pentland, A.: Real-time american sign language recognition from video using hidden markov models. Technical Report MIT TR-375, Media Lab, MIT (1996)Google Scholar
  16. 16.
    Rabiner, L.: A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE 77(1), 257–286 (1989)CrossRefGoogle Scholar
  17. 17.
    Avilés-Arriaga, H., Sucar, L.E.: Dynamic bayesian networks for visual recognition of dynamic gestures. Journal of Intelligent and Fuzzy Systems 12, 243–250 (2002)zbMATHGoogle Scholar
  18. 18.
    Flodin, M.: Signing illustrated: the complete learning guide. The Berkley Publishing Group, New York (1994)Google Scholar
  19. 19.
    Hernandez-Rebollar, J.L., Lindeman, R.W., Kyriakopoulos, N.: A multi-class pattern recognition system for practical finger spelling translation. In: Proc. of IEEE 4th Int’l Conference on Multimodal Interfaces (ICMI 2002), pp. 185–190 (2002)Google Scholar
  20. 20.
    Nissen, S.: Fast artificial neural network library (2000),
  21. 21.
    Hernandez-Rebollar, J.L., Lindeman, R.W., Kyriakopoulos, N.: The acceleglove: a whole hand input device for virtual reality. In: Proc. of ACM Siggraph 2002 - 29th Int’l Conference on Computer Graphics and Interactive Techniques - Sketches and Applications, p. 259 (2002)Google Scholar
  22. 22.
    Kuroda, T., Tabata, Y., Goto, A., Ikuta, H., Murakami, M.: Consumer price data-glove for sign language recognition. In: Proc. of 5th Intl Conf. Disability, Virtual Reality Assoc. Tech., Oxford, UK, pp. 253–258 (2004)Google Scholar
  23. 23.
    Dawnsign: Numbering in American Sign Language: Number Signs for Everyone. DawnSign Press (1998)Google Scholar
  24. 24.
    Ong, S., Ranganath, S.: Automatic sign language analysis: A survey and the future beyond lexical meaning. IEEE Transactions on Pattern Analysis and Machine Intelligence 27, 873–891 (2005)CrossRefGoogle Scholar
  25. 25.
    Su, M.C.: A fuzzy rule-based approach to spatio-temporal hand gesture recognition. IEEE Trans. Systems, Man, and Cybernetics, Part C: Application Rev. 30, 276–281 (2000)CrossRefGoogle Scholar
  26. 26.
    Adamo-Villani, N., Carpenter, E., Arns, L.: An immersive virtual environment for learning sign language mathematics. In: ACM Proceedings of Siggraph 2006 - 33rd International Conference on Computer Graphics and Interactive Techniques - Educators, ACM Siggraph, New York (2006)Google Scholar
  27. 27.
    Valli, C., Lucas, C.: Linguistics of American Sign Language: a Resource Text for ASL Users. Gallaudet University Press, Washington (2002)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Nicoletta Adamo-Villani
    • 1
  • Bedřich Beneš
    • 1
  • Matt Brisbin
    • 1
  • Bryce Hyland
    • 1
  1. 1.Purdue UniversityWest LafayetteUSA

Personalised recommendations