Universal Access in the Information Society

, Volume 6, Issue 4, pp 419–434 | Cite as

Generating American Sign Language animation: overcoming misconceptions and technical challenges

  • Matt Huenerfauth
Long Paper


Misconceptions about the English literacy rates of deaf Americans, the linguistic structure of American Sign Language (ASL), and the suitability of traditional machine translation (MT) technology to ASL have slowed the development of English-to-ASL MT systems for use in accessibility applications. This article traces the progress of a new English-to-ASL MT project targeted to translating texts important for literacy and user-interface applications. These texts include ASL phenomena called “classifier predicates.” Challenges in producing classifier predicates, novel solutions to these challenges, and applications of this technology to the design of user-interfaces accessible to deaf users will be discussed.


American Sign Language Deafness Assistive Technology Natural Language Processing Machine Translation 



American Sign Language


Natural Language Processing


Machine Translation


Graphical User Interface



This work was supported by a grant from the US National Science Foundation (Award #0520798 “SGER: Generating Animations of ASL Classifier Predicates,” Universal Access Program, 2005). Software used in this project has been donated by Siemens UGS Tecnomatix and Autodesk. I would like to thank my collaborators at the Center for Human Modeling and Simulation at the University of Pennsylvania: Liming Zhao, Erdan Gu, and Jan Allbeck. I would also like to thank Mitch Marcus, Martha Palmer, and Norman Badler for their guidance and support during this work.


  1. 1.
    Bindiganavale, R., Schuler, W., Allbeck, J., Badler, N., Joshi, A., Palmer, M.: Dynamically altering agent behaviors using natural language instructions. In: Proceedings of the 4th International Conference on Autonomous Agents, AGENTS 2000, 3–7 June 2000, Barcelona, Catalonia, Spain (2000)Google Scholar
  2. 2.
    Coulter G (ed) Phonetics and phonology: current issues in American Sign Language Phonology. Academic, New York (1993)Google Scholar
  3. 3.
    Elliott, R., Glauert, J., Jennings, V., Kennaway, J.: An overview of the SiGML Notation and SiGML Signing Software System. In: Streiter, O., Vettori, C. (eds.), Proceedings of the Workshop on the Representation and Processing of Signed Languages, 4th International Conference on Language Resources and Evaluation: LREC 2004. 30 May 2004, Lisbon, Portugal, pp. 98–104 (2004)Google Scholar
  4. 4.
    Holt, J.: Demographic stanford achievement test—8th edition for deaf and hard of hearing students: reading comprehension subgroup results (1991)Google Scholar
  5. 5.
    Huenerfauth, M.: A survey and critique of American Sign Language natural language generation and machine translation systems. Technical Report MS-CIS-03–32, Computer and Information Science, University of Pennsylvania (2003)Google Scholar
  6. 6.
    Huenerfauth, M.: A multi-path architecture for machine translation of English text into American Sign Language animation. In: Proceedings of the Student Workshop of the Human Language Technologies conference/North American chapter of the Association for Computational Linguistics annual meeting: HLT/NAACL 2004. Boston, MA, USA (2004)Google Scholar
  7. 7.
    Huenerfauth, M.: Spatial representation of classifier predicates for machine translation into American Sign Language. In: Proceedings of the Workshop on the Representation and Processing of Signed Languages, 4th International Conference on Language Resources and Evaluation: LREC 2004. Lisbon, Portugal (2004)Google Scholar
  8. 8.
    Huenerfauth, M.: Spatial and planning models of ASL classifier predicates for machine translation. In: Proceedings of the 10th international conference on theoretical and methodological issues in machine translation: TMI 2004, Baltimore, MD, USA (2004)Google Scholar
  9. 9.
    Huenerfauth, M.: American Sign Language generation: multimodal NLG with multiple linguistic channels. In: Proceedings of the Association for Computational Linguistics, 43rd Annual Meeting, Student Research Workshop, Ann Arbor, MI, USA (2005)Google Scholar
  10. 10.
    Huenerfauth, M.: American Sign Language spatial representations for an accessible user-interface. In: Proceedings of the 3rd international conference on universal access in human-computer interaction, Las Vegas, NV, USA (2005)Google Scholar
  11. 11.
    Huenerfauth, M.: Representing coordination and non-coordination in an American Sign Language Animation. In: Proceedings of the 7th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2005), Baltimore, MD, USA (2005)Google Scholar
  12. 12.
    Huenerfauth, M: Generating American Sign Language classifier predicates for English-to-ASL Machine Translation, Ph.D. Dissertation, Computer and Information Science, University of Pennsylvania (2006)Google Scholar
  13. 13.
    Liddell, S.: Grammar gesture and meaning in American Sign Language. Cambridge University Press, Cambridge (2003)Google Scholar
  14. 14.
    Liddell, S.: Sources of meaning in ASL classifier predicates. In: Emmorey, K. (eds.) Perspectives on classifier constructions in sign languages. Workshop on Classifier Constructions, La Jolla (2003)Google Scholar
  15. 15.
    Liu, Y.: Interactive reach planning for animated characters using hardware acceleration. Doctoral Dissertation, Computer and Information Science, University of Pennsylvania (2003)Google Scholar
  16. 16.
    Mitchell, R. How many deaf people are there in the United States. Retrieved June 28, 2004 from Gallaudet Research Institute, Graduate School and Professional Programs, Gallaudet University Web site: (2004)
  17. 17.
    Morford, J., MacFarlane, J.: Frequency characteristics of American Sign Language. Sign Lang. Stud. 3(2), 213–225 (2003)CrossRefGoogle Scholar
  18. 18.
    Neidle, C., Kegl, J., MacLaughlin, D., Bahan, B., Lee, R.: The syntax of American Sign Language: functional categories and hierarchical structure. MIT, Cambridge (2000)Google Scholar
  19. 19.
    Sáfár, É., Marshall, I.: The architecture of an English-Text-to-Sign-Languages translation system. In: Angelova, G. (ed.) Recent advances in natural language processing (RANLP). Tzigov Chark, Bulgaria, pp. 223–228 (2001)Google Scholar
  20. 20.
    Wideman, C., Sims, M.: Signing avatars. In: Proceedings of the Technology and Persons with Disabilities Conference, March 15–20, 1999, Los Angeles, CA, USA (1998)Google Scholar
  21. 21.
    Zhao, L., Kipper, K., Schuler, W., Vogler, C., Badler, N., Palmer, M.: A machine translation system from English to American Sign Language. In: Proceedings of the 4th conference of the association for machine translation in the americas on envisioning machine translation in the information future, lecture notes in computer science, 1934, Springer, Heidelberg, London, pp. 54–67 (2000)Google Scholar
  22. 22.
    Zhao, L., Liu, Y., Badler, N.I.: Applying empirical data on upper torso movement to real-time collision-free reach tasks. In: Proceedings of the SAE Digital Human Modeling Conference, Iowa City (2005)Google Scholar

Copyright information

© Springer-Verlag 2007

Authors and Affiliations

  1. 1.Department of Computer ScienceQueens College, The City University of New YorkFlushingUSA

Personalised recommendations