Machine Translation

, Volume 13, Issue 1, pp 81–106 | Cite as

The Challenges of Cross-Modal Translation: English-to-Sign-Language Translation in the Zardoz System

  • Tony Veale
  • Alan Conway
  • BrÓna Collins


The sign languages used by deaf communities around the world represent a linguistic challenge that natural-language researchers in AI have only recently begun to take up. This challenge is particularly relevant to research in Machine Translation (MT), as natural sign languages have evolved in deaf communities into efficient modes of gestural communication, which differ from English not only in modality but in grammatical structure, exploiting a higher dimensionality of spatial expression. In this paper we describe Zardoz, an on-going AI research system that tackles the cross-modal MT problem, translating English text into fluid sign language. The paper presents an architectural overview of Zardoz, describing its central blackboard organization, the nature of its interlingual representation, and the major components which interact through this blackboard both to analyze the verbal input and generate the corresponding gestural output in one of a number of sign variants.

sign-language space gesture cross-modal translation metaphor 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Adriaens, G. and S.L. Small: 1988, ‘Word Expert Parsing Revisited in a Cognitive Science Perspective’, in S.L. Small, G.W. Cottrell and M.K. Tanenhaus (eds), Lexical Ambiguity, Morgan Kaufmann, San Mateo, CA, pp. 58–79.Google Scholar
  2. Chandioux, J.: 1976, ‘MÉTÉO: un système opérationnel pour la traduction automatique des bulletins météorologiques destinés au grand public’, Meta 21, 127–133.Google Scholar
  3. Chandioux, J.: 1989, ‘Météo: 100 Million Words Later’, in D.L. Hammond (ed.), AmericanTranslators Association Conference 1989: Coming of Age, Learned Information, Medford, NJ, pp. 12–20.Google Scholar
  4. Conway, A. and T. Veale: 1994, ‘A Linguistic Approach to Sign Language Synthesis’, in HumanComputer Interface Conference (HCI), Glasgow, pp. 35–45.Google Scholar
  5. Conway, A. and T. Veale: 1995, ‘Building Signs: Representing Space and Structure in Automatic Sign Synthesis’, in 5th International Cognitive Linguistics Conference, Albuquerque, NM, pp. 89–95.Google Scholar
  6. Cunningham, P. and T. Veale: 1991, ‘Organizational Issues Arising from the Integration of the Concept Network and Lexicon in a Text Understanding System’, in 12th International JointConference on Artificial Intelligence, Sydney, Australia, pp. 1009–1015.Google Scholar
  7. Emmorey, K.: 1995, ‘Interactions between Processing Spatial Information in Linguistic and Non-Linguistic Domains’, in 5th International Cognitive Linguistics Conference, Albuquerque, NM, pp. 34–40.Google Scholar
  8. Grushkin, D.: 1995, ‘Metaphorical Expressions of Anger in ASL as a Window on the Culture of the American Deaf Community’, in 5th International Cognitive Linguistics Conference, Albuquerque, NM, pp. 12–18.Google Scholar
  9. Holden, E.J. and G.G. Roy: 1992, ‘The Graphical Translation of English Text into Signed English in the Hand Sign Translator System’, Computer Graphics Forum 113, 357–366.Google Scholar
  10. Klima, E. and U. Bellugi: 1979, The Signs of Language. Harvard University Press, Cambridge, MA.Google Scholar
  11. Lakoff, G. and M. Johnson: 1980, Metaphors We Live By. University of Chicago Press, Chicago.Google Scholar
  12. Lee, J. and T.L. Kunii: 1992, ‘Visual Translation from Native Language to Sign Language’, in IEEEWorkshop on Visual Languages, Seattle, WA, pp. 45–53.Google Scholar
  13. Liddell, S.K.: 1980, American Sign Language Syntax. Mouton, The Hague.Google Scholar
  14. Liddell, S.K. and R.E. Johnson: 1986, ‘American Sign Language Compound Formation Processes, Lexicalization and Phonological Remnants’, Natural Language and Linguistic Theory 4, 445–513.Google Scholar
  15. Lyons, J.: 1977, Semantics. Cambridge University Press, London.Google Scholar
  16. Mellish, C.S.: 1989, ‘Some Chart-Based Techniques for Parsing Ill-Formed Input’, in 27th AnnualMeeting of the Association for Computational Linguistics, Vancouver, British Columbia, pp. 89–96.Google Scholar
  17. Mitamura, T., E.H. Nyberg, and J.G. Carbonell: 1991, ‘An Efficient Interlingua Translation System for Multi-lingual Document Production’, in Machine Translation Summit III, Washington, DC, pp. 15–23.Google Scholar
  18. Padden, C.A. and D.M. Perlmutter: 1987, ‘American Sign Language and the Architecture of Phonological Theory’, Natural Language and Linguistic Theory 5, 335–375.Google Scholar
  19. Patten, T. and J. Hartigan: 1993, ‘Automatic Translation of English to American Sign Language’, in Proceedings of the 1993 National Conference on Deafness, Columbus, OH, pp. 55–65.Google Scholar
  20. Sandler, W.: 1989, Phonological Representation of the Sign: Linearity and Non-Linearity inAmerican Sign Language. Foris Publications, Providence, RI.Google Scholar
  21. Shieber, S.: 1986, An Introduction to Unification-Based Approaches to Language. Center for the Study of Language and Information, Stanford, CA.Google Scholar
  22. Veale, T. and B. Collins: 1996, ‘Space, Schematization and Metaphor in Sign: Sign Language Translation in the Zardoz System’, in Expanding MT Horizons: Proceedings of the Second Conferenceof the Association for Machine Translation in the Americas, Montréal, Québec, pp. 150–158.Google Scholar
  23. Veale, T. and A. Conway: 1994, ‘Cross-Modal Comprehension in Zardoz, An English to Sign-Language Translation System’, in 4th International Workshop on Natural Language Generation, Kennebunkport, ME, pp. 67–72.Google Scholar
  24. Veale, T. and P. Cunningham: 1992, ‘Competitive Hypothesis Resolution in TWIG: A Blackboard-Driven Text-Understanding System’, in 10th European Conference on Artificial Intelligence, Vienna, Austria, pp. 105–117.Google Scholar
  25. Veale, T. and M.T. Keane: 1992a, ‘Conceptual Scaffolding: Using Metaphors to Build Knowledge Structures’, in 10th European Conference on Artificial Intelligence, Vienna, Austria, pp. 93–100.Google Scholar
  26. Veale, T. and M.T. Keane: 1992b, ‘Conceptual Scaffolding: A Spatially Founded Meaning Representation for Metaphor Comprehension’, Computational Intelligence 83, 494–519.Google Scholar
  27. Veale, T. and B. Smyth: 1992, ‘Krell: Knowledge Representation Entry-Level Language, the User Guide Version 1.0’, Hitachi Dublin Laboratory Technical Report HDL-TR-92–051.Google Scholar
  28. Weaver, W.: 1949, ‘Translation’, reprinted in W. Locke and A. Booth (eds), Machine Translation of Languages, 1955, MIT Press, Cambridge, MA, pp. 1–15.Google Scholar
  29. Werhli, E.: 1992, ‘The IPS System’, in Proceedings of the fifteenth [sic] International Conferenceon Computational Linguistics, Actes du quinzième colloque international en linguistiqueinformatique: COLING-92, Nantes, pp. 870–874.Google Scholar
  30. Werhli, E.: 1996, ‘The ITSVOX System’, in Expanding MT Horizons: Proceedings of the SecondConference of the Association for Machine Translation in the Americas, Montréal, Québec, pp. 120–127.Google Scholar
  31. Wilks, Y.: 1975, ‘A Preferential, Pattern-Seeking, Semantics for Natural Language Inference’, Artificial Intelligence 6, 53–74.Google Scholar
  32. Yngve, V.: 1957, ‘A Framework for Syntactic Translation’. Mechanical Translation 43, 59–65.Google Scholar
  33. Yamada, K.: 1998, ‘A Controlled Skip Parser’, this volume.Google Scholar

Copyright information

© Kluwer Academic Publishers 1998

Authors and Affiliations

  • Tony Veale
    • 1
  • Alan Conway
    • 2
  • BrÓna Collins
    • 3
  1. 1.School of Computer ApplicationDublin City UniversityGlasnevinIreland
  2. 2.Iona TechnologiesDublinIreland
  3. 3.Department of Computer ScienceTrinity CollegeDublinIreland

Personalised recommendations