Skip to main content
SpringerLink
Log in

Neural machine translation from text to sign language

  • Long Paper
  • Published:
Universal Access in the Information Society Aims and scope Submit manuscript

Abstract

The paper describes an ongoing project aimed at developing a neural machine translation approach to translating text into sign language, with the translation result presented by a realistic three-dimensional avatar. The approach can be applied to translate texts, books, and Internet pages, improving access to information for deaf individuals and contributing to the social, educational, and labor inclusion of these citizens. The paper elaborates on four fundamental issues related to the approach: (i) the establishment of a written representation of the sign language; (ii) the construction of a parallel corpus involving text from an oral language and the written representation of sign language; (iii) the establishment of a neural machine translation model to perform the translation; (iv) the visual presentation of the translation by a signing avatar. Although focused on translating Brazilian Portuguese into Brazilian Sign Language, the concepts discussed in the paper are general enough to be applied to other written-signed language pairs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)

  2. Brour, M., Benabbou, A.: Atlaslang mts 1: Arabic text language into arabic sign language machine translation system. Procedia Computer Science 148:236–245. https://doi.org/10.1016/j.procs.2019.01.066, https://www.sciencedirect.com/science/article/pii/S1877050919300699, The second international conference in intelligent computing in data sciences – ICDS2018 (2019)

  3. Bungeroth, J., Ney, H.: Statistical sign language translation. In: Streiter, O., Vettori, C. (eds) Proceedings of the LREC2004 workshop on the representation and processing of sign languages: from signwriting to image processing. information techniques and their implications for teaching, documentation and communication. European Language Resources Association (ELRA), Lisbon, Portugal, pp 105–108 (2004)

  4. Bungeroth, J., Stein, D., Dreuw, P., et al.: A german sign language corpus of the domain weather report. In: Proceedings of the fifth international conference on language resources and evaluation (LREC’06). European Language Resources Association (ELRA), Genoa, Italy (2006)

  5. Capovilla, F.C., Rahael, W.D., Temoteo, J.G., et al.: Dicionário Da Língua De Sinais Do Brasil - A Libras Em Suas Mãos. Edusp, São Paulo (2017)

    Google Scholar 

  6. Clark, A.: How to write American sign language. ASLwrite, Burnsville, Minnesota (2012)

    Google Scholar 

  7. Cormier, K., Fenlon, J., Gulamani, S., et al.: BSL corpus annotation conventions version 2.023. Deafness Cognition and Language (DCAL) Research Centre, University College London, Technical report (2015)

  8. Dabre, R., Chu, C., Kunchukuttan, A.: A survey of multilingual neural machine translation. ACM Comput. Surv. (CSUR) 53(5), 1–38 (2020)

    Article  Google Scholar 

  9. Devlin, J., Chang, M., Lee, K., et al.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

  10. ELAN: ELAN (version 6.4). The language archive, https://archive.mpi.nl/tla/ela, computer software (2022)

  11. Elliott, R., Glauert, J.R.W., Kennaway, J.R., et al.: The development of language processing support for the visicast project. In: Proceedings of the fourth international acm conference on assistive technologies. Association for Computing Machinery, New York, NY, USA, Assets ’00, p 101-108, https://doi.org/10.1145/354324.354349, (2000)

  12. Estelita, B.M.: Elis – escrita das línguas de sinais: proposta teórica e verificação prática. PhD thesis, Universidade Federal de Santa Catarina, Florianópolis, Santa Catarina, Brazil (2008)

  13. Gómez, S.E., McGill, E., Saggion, H.: Syntax-aware transformers for neural machine translation: the case of text to sign gloss translation. In: Proceedings of the 14th workshop on building and using comparable corpora (BUCC 2021), pp 18–27 (2021)

  14. Green, T.: The necessity of syntax markers: two experiments with artificial languages. J. Verbal Learn. Verbal Behav. 18(4), 481–496 (1979). https://doi.org/10.1016/S0022-5371(79)90264-0

    Article  Google Scholar 

  15. Hanke, T.: Hamnosys in a sign language generation context. In: Schulmeister, R., Reinitzer, H.: (eds) Progress in Sign Language Research / Fortschritte in der Gebärdensprachforschung. Signum, p 249–266 (2002)

  16. Johnson, R.E., Liddell, S.K.: A segmental framework for representing signs phonetically. Sign Lang. Stud. 11(3), 408–463 (2011)

    Article  Google Scholar 

  17. Johnson, R.E., Liddell, S.K.: Toward a phonetic representation of hand configuration: the fingers. Sign Lang. Stud. 12(1), 5–45 (2011)

    Article  Google Scholar 

  18. Johnson, R.E., Liddell, S.K.: Toward a phonetic representation of signs: sequentiality and contrast. Sign Lang. Stud. 11(2), 241–274 (2011)

    Article  Google Scholar 

  19. Johnson, R.E., Liddell, S.K.: Toward a phonetic representation of hand configuration: the thumb. Sign Lang. Stud. 12(2), 316–333 (2012)

    Article  Google Scholar 

  20. Johnston, T.: Transcription and glossing of sign language texts: examples from Auslan (Australian Sign Language). Int. J. Sign Linguist. 2(1), 3–28 (1991)

    Google Scholar 

  21. Johnston, T.: Corpus linguistics and signed languages: no lemmata, no corpus. In: Crasborn, O., Efthimiou, E., Hanke, T., et al. (eds.) LREC 6–3rd workshop on the representation and processing of sign languages: construction and exploitation of signed language corpora, pp. 82–87. Marrakech, Morocco (2008)

    Google Scholar 

  22. Johnston, T.: From archive to corpus: transcription and annotation in the creation of signed language corpora. In: 22nd Pacific Asian conference on language, information, and computation, pp 16–29 (2008b)

  23. Johnston, T.: Auslan corpus annotation guidelines. Technical report, Macquarie University (Sydney) - La Trobe University (Melbourne) (2019)

  24. Kahlon, N.K., Singh, W.: Machine translation from text to sign language: a systematic review. Univ. Access Inf. Soc. (2021). https://doi.org/10.1007/s10209-021-00823-1

    Article  Google Scholar 

  25. Kamata, K., Yoshida, T., Watanabe, M., et al.: An approach to japanese-sign language translation system. In: Conference proceedings., IEEE international conference on systems, man and cybernetics, pp 1089–1090 vol.3, https://doi.org/10.1109/ICSMC.1989.71466 (1989)

  26. Kim, Y., Denton, C., Hoang, L., et al.: Structured attention networks. arXiv preprint arXiv:1702.00887 (2017)

  27. Koehn, P., Federico, M., Shen, W., et al.: Open source toolkit for statistical machine translation: factored translation models and confusion network decoding. In: CLSP Summer Workshop Final Report WS-2006, Johns Hopkins University (2007)

  28. Kouremenos, D., Ntalianis, K., Kollias, S.: A novel rule based machine translation scheme from Greek to Greek sign language: production of different types of large corpora and language models evaluation. Comput. Speech Lang. 51, 110–135 (2018). https://doi.org/10.1016/j.csl.2018.04.001

    Article  Google Scholar 

  29. Liddell, S.: Grammar, gesture and meaning in American sign language. Cambridge University Press, England (2003)

    Book  Google Scholar 

  30. Luong, M.T., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025 (2015)

  31. Luqman, H., Mahmoud, S.A.: Automatic translation of Arabic text-to-Arabic sign language. Univ. Access Inf. Soc. 18(4), 939–951 (2019). https://doi.org/10.1007/s10209-018-0622-8

    Article  Google Scholar 

  32. McCleary, L., Viotti, E., Leite, T.A.: Descrição das línguas sinalizadas: a questão da transcrição dos dados. ALFA 54(1), 265–289 (2010)

    Google Scholar 

  33. Morrissey, S., Way, A.: An example-based approach to translating sign language. In: Workshop on example-based machine translation, Phuket, Thailand, pp 109–116, https://aclanthology.org/2005.mtsummit-ebmt.14 (2005)

  34. Naert, L., Larboulette, C., Gibet, S.: A survey on the animation of signing avatars: from sign representation to utterance synthesis. Comput. Graph. 92, 76–98 (2020). https://doi.org/10.1016/j.cag.2020.09.003

    Article  Google Scholar 

  35. Núñez-Marcos, A., Perez-de Viñaspre, O., Labaka, G.: A survey on sign language machine translation. Expert Syst. Appl. 213, 118993 (2022)

    Article  Google Scholar 

  36. Papineni, K., Roukos, S., Ward, T., et al.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp 311–318 (2002)

  37. Raffel, C., Shazeer, N., Roberts, A., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 (2019)

  38. San-Segundo, R., Barra-Chicote, R., D’Haro, L., et al.: A spanish speech to sign language translation system for assisting deaf-mute people. In: INTERSPEECH 2006 and 9th international conference on spoken language processing, INTERSPEECH 2006 - ICSLP, https://doi.org/10.21437/Interspeech.2006-420 (2006)

  39. San-Segundo, R., Montero, J.M., Córdoba, R., et al.: Design, development and field evaluation of a Spanish into sign language translation system. Pattern Anal. Appl. 15(2), 203–224 (2012). https://doi.org/10.1007/s10044-011-0243-9

    Article  MathSciNet  Google Scholar 

  40. Saunders, B., Camgoz, N.C., Bowden, R.: Progressive transformers for end-to-end sign language production. arXiv preprint arXiv:2004.14874 (2020)

  41. Shogun: Shogun software. https://www.vicon.com/software/shogun/ (2022)

  42. Sountsov, P., Sarawagi, S.: Length bias in encoder decoder models and a case for global conditioning. arXiv preprint arXiv:1606.03402 (2016)

  43. Stahlberg, F.: Neural machine translation: a review. J. Artif. Intell. Res. 69, 343–418 (2020)

    Article  MathSciNet  Google Scholar 

  44. Stokoe, W.C.: Sign language structure: an outline of the visual communication system of the American deaf. Studies in Linguistics: Occasional Papers 8 Revised 1978 (1960)

  45. Stoll, S., Camgoz, N.C., Hadfield, S., et al.: Text2sign: towards sign language production using neural machine translation and generative adversarial networks. Int. J. Comput. Vision 128(4), 891–908 (2020). https://doi.org/10.1007/s11263-019-01281-2

    Article  Google Scholar 

  46. Sutton, V.: Lessons in SignWriting—Textbook, 4th edn. SignWriting Press, USA (2014)

    Google Scholar 

  47. Tan, X., Qin, T., Soong, F., et al.: A survey on neural speech synthesis. arXiv (2021). https://doi.org/10.48550/ARXIV.2106.15561

    Article  Google Scholar 

  48. Vaswani, A., Shazeer, N., Parmar, N., et al.: Attention is all you need. In: Guyon, I., Von Luxburg, U., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in neural information processing systems. Curran Associates Inc, Red Hook (2017)

    Google Scholar 

  49. Veale, T., Conway, A.: Cross-modal comprehension in zardoz, an english to sign-language translation system. In: 4th International Workshop on Natural Language Generation, pp. 67–72. Kennebunkport, ME (1994)

  50. Veale, T., Conway, A., Collins, B.: The challenges of cross-modal translation: English-to-sign-language translation in the zardoz system. Mach. Transl. 13(1), 81–106 (1998). https://doi.org/10.1023/A:1008014420317

    Article  Google Scholar 

  51. Will, A.D., Martino, J.M.D., Bezerra, J.: An optimized marker layout for 3D facial motion capture. In: Livesu, M., Pintore, G., Signoroni, A. (eds) Smart Tools and Apps for Graphics—Eurographics Italian Chapter conference. The Eurographics Association, Brescia, Italy, https://doi.org/10.2312/stag.20181303(2018)

  52. Wu, C.H., Su, H.Y., Chiu, Y.H., et al.: Transfer-based statistical translation of Taiwanese sign language using PCFG. ACM Trans. Asian Lang. Inf. Process. 6(1), 1-es (2007). https://doi.org/10.1145/1227850.1227851

    Article  Google Scholar 

  53. Wu, Y., Schuster, M., Chen, Z., et al.: Google’s neural machine translation system: bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016)

  54. Zhao, L., Kipper, K., Schuler, W., et al.: A machine translation system from English to American sign language. In: Fourth conference of the association for machine translation in the Americas: Technical Papers, pp. 54–67. Springer-Verlag, Berlin, Heidelberg (2000)

  55. Zoph, B., Yuret, D., May, J., et al.: Transfer learning for low-resource neural machine translation. arXiv preprint arXiv:1604.02201 (2016)

Download references

Funding

This study is partly financed by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001 Grant no 88887.091672/2014-01, the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq) Grant no 458691/2013-5, the Financiadora de Estudos e Projetos (Finep) Grant no 2778/20, and the Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP) Grant no 2021/02365-2.

Author information

Author notes

  1. Ivani Rodrigues Silva, Janice Gonçalves Temoteo Marques, Antonielle Cantarelli Martins, Enzo Telles Poeta, Dener Stassun Christinele and João Pedro Araújo Ferreira Campos have contributed equally to this work.

Authors and Affiliations

  1. Faculdade de Engenharia Elétrica e de Computação, Universidade Estadual de Campinas, Avenida Albert Einstein, 400, Campinas, São Paulo, 13083-852, Brazil

    José Mario De Martino, Antonielle Cantarelli Martins, Enzo Telles Poeta & João Pedro Araújo Ferreira Campos

  2. Faculdade de Ciência Médicas, Universidade Estadual de Campinas, Rua Tessália Vieira de Camargo, 126, Campinas, São Paulo, 13083-887, Brazil

    Ivani Rodrigues Silva & Janice Gonçalves Temoteo Marques

  3. ShowCase PRO, Av. Antônio Artioli, 570, Campinas, São Paulo, 13049-253, Brazil

    Dener Stassun Christinele

Authors
  1. José Mario De Martino
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ivani Rodrigues Silva
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Janice Gonçalves Temoteo Marques
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Antonielle Cantarelli Martins
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Enzo Telles Poeta
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Dener Stassun Christinele
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. João Pedro Araújo Ferreira Campos
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study’s conception and design. All authors participated in the production of all versions of the manuscript and commented on previous versions of the manuscript. All authors read and approved the final manuscript. As the Corresponding Author, I state that all authors contributed equally to the manuscript, writing and reviewing the text and preparing the figures and table.

Corresponding author

Correspondence to José Mario De Martino.

Ethics declarations

Conflict of interest

On behalf of all the authors, the corresponding author states that there are no competing interests that might be perceived to influence the results and discussion reported in this paper. The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

De Martino, J.M., Silva, I.R., Marques, J.G.T. et al. Neural machine translation from text to sign language. Univ Access Inf Soc (2023). https://doi.org/10.1007/s10209-023-01018-6

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10209-023-01018-6

Keywords

Search

Navigation