Skip to main content

Advertisement

SpringerLink
Log in

Signing avatars: making education more inclusive

  • Long Paper
  • Published:
Universal Access in the Information Society Aims and scope Submit manuscript

Abstract

In Brazil, there are approximately 9.7 million inhabitants who are deaf or hard of hearing. Moreover, about 30% of the Brazilian deaf community is illiterate in Brazilian Portuguese due to difficulties to offer deaf children an inclusive environment based on bilingual education. Currently, the prevailing teaching practice depends heavily on verbal language and on written material, making the inclusion of the deaf a challenging task. This paper presents the author’s approach for tackling this problem and improving deaf students’ accessibility to written material in order to help them master Brazilian Portuguese as a second language. We describe an ongoing project aimed at developing an automatic Brazilian Portuguese-to-Libras translation system that presents the translated content via an animated virtual human, or avatar. The paper describes the methodology adopted to compile a source language corpus having the deaf student needs in central focus. It also describes the construction of a parallel Brazilian Portuguese/Brazilian Sign Language (Libras) corpus based on motion capture technology. The envisioned translation architecture includes the definition of an Intermediate Language to drive the signing avatar. The results of a preliminary assessment of signs intelligibility highlight the application potential.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Almohimeed, A., Wald, M,. Damper, R.I.: Arabic text to arabic sign language translation system for the deaf and hearing-impaired community. In: Proceedings of the Second Workshop on Speech and Language Processing for Assistive Technologies, Association for Computational Linguistics, Stroudsburg, PA, USA, SLPAT’11, pp. 101–109 (2011)

  2. Amaral, W.M.: Sistema de transcrição da língua brasileira de sinais voltado à produção de conteúdo sinalizado por avatares 3D. PhD thesis, Faculdade de Engenharia Elétrica e de Computação—Universidade Estadual de Campinas (2012)

  3. Amaral, W.M., De Martino, J.M.: Towards a transcription system of sign language for 3D virtual agents. In: Tarek, Sobh (ed.) Innovations in Computing Sciences and Software Engineering, pp. 85–90. Springer, Netherlands (2010)

    Chapter  Google Scholar 

  4. Amaral, W.M., Angare, L., De Martino, J.M.: Sign Language 3D Virtual Agent. In: Proceedings of the 5th International Multi-Conference on Society, Cybernetics and Informatics (IMSCI 2011), International Institute of Informatics and Systemics, Orlando, Florida, vol. I, pp. 93–97 (2011)

  5. Baker, M.: Corpus linguistics and translation studies: implications and applications. In: Baker, M., Francis, G., Tognini-Bonelli, E. (eds.) Text and Technology: In Honour of John Sinclair, pp. 233–248. John Benjamins, Amsterdam (1993)

    Chapter  Google Scholar 

  6. Brasil: Lei no. 9.394. In: Diário Oficial União, Imprensa Oficial República Federativa do Brasil, Brasília, DF, pp. 1–9 (1996)

  7. Brasil: Lei no. 10.436. In: Diário Oficial União, Imprensa Oficial República Federativa do Brasil, Brasília, DF, p. 23 (2002)

  8. Brasil: Decreto no. 5.626. In: Diário Oficial União, Imprensa Oficial República Federativa do Brasil, Brasília, DF, pp. 28–30 (2005)

  9. Breda, W.L., Caon, D.R.S., Brito, P.H.S., Tavares, O.L., Meneses, C., Coradine, L.C., Albuquerque, F.C.: SOTAC: a software for knowledge-based automatic translation. In: Proceedings of the 9th IFIP World Conference on Computers in Education, Springer, Bento Gonçalves, RS, Brazil, vol. 1, pp. 1–10 (2009)

  10. Brito, P.H.S., Franco, N., Coradine, L.C.: Falibras: Uma Ferramenta Flexível para Promover Acessibilidade de Pessoas Surdas. In: Anais del XVII Congreso Internacional de Informática Educativa (TISE 2012), Universidad de Chile, Santiago, Chile, pp. 87–96 (2012)

  11. Bungeroth, J., Ney, H.: Statistical sign language translation. In: Workshop on Representation and Processing of Sign Languages, LREC vol. 4, 105–108 (2004)

    Google Scholar 

  12. Capovilla, F.C., Raphael, W.D.: Dicionário Enciclopédico Ilustrado Trilíngue da Língua de Sinais Brasileira \(Libras\), 3rd edn. EDUSP, São Paulo, SP, Brazil (2008)

  13. Capovilla, F.C., Raphael, W.D., Mauricio, A.C.: Novo Deit-libras: Dicionário Enciclopédico Ilustrado Trilingue da Língua de Sinais Brasileira \(Libras\). EDUSP, São Paulo, SP (2009)

  14. Coradine, L.C., Albuquerque, F.C., Brito, P.H.S., Silva, R.L., Silva, T.F.L.: Sistema Falibras: interpretação animada, em Libras, de palavras e expressões em português. In: Anais do III Congresso Ibero-Americano de Informática na Educação Especial (CIIEE 2002), Fortaleza/CE, Brazil, vol. 1 (2002)

  15. Coradine, L.C., Tavares, O.L., Albuquerque, F.C., Ribeiro, M.N., Breda, W.L., Calado, I.A.A.R., Smarsaro, A.D.: Sistema Falibras: Um intérprete virtual como ferramenta de apoio pedagógico à educação de surdos. In: Anais do VII Congresso Ibero-Americano de Informática Educativa Especial (CIIEE 2007), Mar del Plata-Argentina, vol. 1, pp. 1–10 (2007)

  16. Côrrea, Y., Vieira, M.C., Santarosa, L.M.C., Biasuz, M.C.V.: Aplicativos de tradução automática como facilitadores da comunicaçã entre surdos e ouvintes. In: Santarosa, L.M.C., Conforto, D., Vieira, M.C. (eds.) Tecnologia e Acessibilidade: Passos em direção à inclusão escolar e sociodigital. Evangraf, Porto Alegre (2014)

    Google Scholar 

  17. Cox, S., Lincoln, M., Tryggvason, J., Nakisa, M., Wells, M., Tutt, M., Abbott, S.: Tessa, a system to aid communication with deaf people. In: Proceedings of the Fifth International ACM Conference on Assistive Technologies, ACM, pp. 205–212 (2002)

  18. Darwiche, A.: Bayesian networks. Commun. ACM 53(12), 80–90 (2010)

    Article  MATH  Google Scholar 

  19. De Martino, J.M., Costa, P.D.P., Benetti, A., Rosa, L.A., Kumada, K.M.O., Silva, I.R.: Building a brazilian portuguese-brazilian sign language parallel corpus using motion capture data. In: Proceedings of Workshop of Corpora and Tools for Processing Corpora, pp. 56-63 (2016)

  20. Editora Moderna (Org): Projeto Pitanguá, 1st edn. Moderna, São Paulo (2005)

  21. Elliott, R., Glauert, J.R., Kennaway, J., Marshall, I., Safar, E.: Linguistic modelling and language-processing technologies for Avatar-based sign language presentation. Univers. Access Inf. Soc. 6(4), 375–391 (2008)

    Article  Google Scholar 

  22. Felipe, T.A.: Introdução à Gramática de Libras. Ministério da Educação e do Desporto. Secretaria de Educação Especial. Educação especial: Deficiência Auditiva, Brasília, DF (1997)

  23. Forster, J., Schmidt, C., Hoyoux, T., Koller, O., Zelle, U., Piater, J.H., Ney, H.: RWTH-PHOENIX-weather: a large vocabulary sign language recognition and translation corpus. In: LREC, pp. 3785–3789 (2012)

  24. Gibet, S., Courty, N., Duarte, K., Naour, T.L.: The SignCom system for data-driven animation of interactive virtual signers: methodology and evaluation. ACM Trans. Interact. Intell. Syst. 1(1), 6:1–6:23 (2011)

    Article  Google Scholar 

  25. Goés, M.C.R.d., Tartuci, D.: Alunos surdos e experiências de letramento. In: Fernades E, Dorzial A, Lodi ACB (eds.) Letramento, Bilinguismo e Educação de Surdos, 2nd edn. Mediação Editora, Porto Alegre, pp. 289–302 (2015)

  26. Gottlieb, G.L., Song, Q., Almeida, G.L., Hong, D.A., Corcos, D.: Directional control of planar human arm movement. J. Neurophysiol 78, 2985–2998 (1997)

    Google Scholar 

  27. Hirashima, M., Kudo, K., Watarai, K., Ohtsuki, T.: Control of 3D Limb Dynamics in unconstrained overarm throws of different speeds performed by skilled baseball players. J. Neurophysiol. 97(1), 680–691 (2007)

    Article  Google Scholar 

  28. Huenerfauth, M.: A multi-path architecture for machine translation of English text into American sign language animation. In: Proceedings of the Student Research Workshop at HLT-NAACL 2004, Association for Computational Linguistics, Stroudsburg, PA, USA, HLT-SRWS ’04, pp. 25–30 (2004)

  29. IBGE: Censo Demográfico 2010—Características gerais da população, religião e pessoas com deficiência. Instituto Brasileiro de Geografia e Estatística, Ministério do Planejamento, Orçamento e Gestão e Instituto Brasileiro de Geografia e Estatística, Rio de Janeiro, Brasil (2010)

  30. Imamura, K., Okuma, H., Watanabe, T., Sumita, E.: Example-based machine translation based on syntactic transfer with statistical models. In: Proceedings of COLING, pp. 99–105 (2004)

  31. Johnson, R.E., Liddell, S.K.: Toward a phonetic representation of hand configuration: the fingers. Sign Lang. Stud. 12(1), 5–45 (2011)

    Article  Google Scholar 

  32. Johnson, R.E., Liddell, S.K.: A segmental framework for representing signs phonetically. Sign Lang. Stud. 11(3), 408–463 (2011)

    Article  Google Scholar 

  33. Johnson, R.E., Liddell, S.K.: Toward a phonetic representation of signs: sequentiality and contrast. Sign Lang. Stud. 11(2), 241–274 (2011)

    Article  Google Scholar 

  34. Johnson, R.E., Liddell, S.K.: Toward a phonetic representation of hand configuration: the thumb. Sign Lang. Stud. 12(2), 316–333 (2012)

    Article  Google Scholar 

  35. Liddell, S.K., Johnson, R.E.: American sign language: the phonological base. Sign Lang. Stud. 64, 195–278 (1989)

    Article  Google Scholar 

  36. Linde-Usiekniewicz, J., Czajkowska-Kisil, M., Łacheta, J., Rutkowski, P.: A corpus-based dictionary of polish sign language (PJM). In: Proceedings of the XVI EURALEX International Congress: The User in Focus (2014)

  37. Lira, G.A., de Souza, T.A.F.: Dicionário da língua brasileira de sinais. http://www.acessobrasil.org.br/libras/ (2008). Accessed August 2016 2008

  38. Lu, P., Huenerfauth, M.: Collecting a motion-capture corpus of American sign language for data-driven generation research. In: Proceedings of the NAACL HLT 2010 Workshop on Speech and Language Processing for Assistive Technologies, Association for Computational Linguistics, Stroudsburg, PA, USA, SLPAT ’10, pp. 89–97 (2010)

  39. Marotta, J.J., Medendorp, W.P., Crawford, J.D.: Kinematic rules for upper and lower arm contributions to grasp orientation. J. Neurophysiol. 90(6), 3816–3827 (2003)

    Article  Google Scholar 

  40. Martin, P.J.M., Belhe, S., Mudliar, S., Kulkarni, M., Sahasrabudhe, S.: An Indian Sign Language (ISL) corpus of the domain disaster message using Avatar. In: Proceedings of the Third International Symposium in Sign Language Translations and Technology (SLTAT-2013), pp. 1–4 (2013)

  41. Massó, G., Badia, T.: Dealing with sign language morphemes in statistical machine translation. In: 4th Workshop on the Representation and Processing of Sign Languages: Corpora and Sign Language Technologies, Valletta, Malta, pp. 154–157 (2010)

  42. McCleary, L., Viotti, E.: Transcrição de dados de uma língua sinalizada: um estudo piloto da transcrição de narrativas na língua de sinais brasileira (LSB). In: Salles, H. (ed.) Bilinguismo e surdez: Questões linguísticas e educacionais, pp. 73–96. Cânone Editorial, Goiania (2007)

  43. McCracken, D.D., Reilly, E.D.: Backus-Naur form (BNF). In: Ralston, A., Reilly, E.D., Hemmendinger, D. (eds.) Encyclopedia of Computer Science, 4th edn, pp. 129–131. Wiley, Chichester (2003)

    Google Scholar 

  44. Meir, I., Sandler, W., Padden, C., Aronoff, M.: Emerging Sign Languages, vol. 2. Oxford University Press, Oxford, pp. 267–280 (Chapter 18) (2010)

  45. Morrissey, S., Way, A.: An example-based approach to translating sign language. In: Second Workshop on Example-Based Machine Translation, Dublin City University, vol. 16, pp. 109–116 (2005)

  46. New London Group: A pedagogy of multiliteracies: designing social futures. In: Cope, B., Kalantzis, M. (eds.) Multiliteracies: Literacy Learning and the Design of Social Futures, pp. 9–38. Macmillan, South Yarra (2000)

    Google Scholar 

  47. Offutt, J.: Quality attributes of web software applications. IEEE Softw. 19(2), 25–32 (2002)

    Article  Google Scholar 

  48. Othman, A., Jemni, M.: Statistical sign language machine translation: from English written text to American sign language gloss. Int. J. Comput. Sci. Issues 8(3), 65–73 (2011)

    Google Scholar 

  49. Pereira, M.C.d.C., Choi, D., Vieira, M.I., Gaspar, P., Nakasato, R.: Libras: conhecimento além dos sinais, 1st edn. Pearson Prentice Hall, São Paulo (2011)

  50. Porta, J., López-Colino, F., Tejedor, J., Colás, J.: A rule-based translation from written Spanish to Spanish Sign Language glosses. Comput. Speech Lang. 28(3), 788–811 (2014)

    Article  Google Scholar 

  51. Probst, K., Levin, L., Peterson, E., Lavie, A., Carbonell, J.: MT for minority languages using elicitation-based learning of syntactic transfer rules. Mach. Transl. 17(4), 245–270 (2002)

    Article  Google Scholar 

  52. Reily, L.H.: As imagens: o lúdico e o absurdo no ensino de arte para pré-escolares surdos. In: Silva, I.R., Kauchakje, S., Gesueli, Z.M. (eds.) Cidadania, surdez e linguagem: desafios e realidades, pp. 161–192. Plexus Editora, Sã Paulo (2003)

    Google Scholar 

  53. Rojo, R.: Gêneros discursivos do Círculo de Bakhtin e multiletramentos. In: Rojo, R.H.R. (ed) Escol@ conectada: os multiletramentos e as TICs, Parábola Editorial, São Paulo, pp. 13–36 (2013)

  54. San-Segundo, R., Montero, J.M., Córdoba, R., Sama, V., Fernández, F., D’Haro, L.F., López-Ludeña, V., Sánchez, D., García, A.: Design, development and field evaluation of a Spanish into sign language translation system. Pattern Anal. Appl. 15(2), 203–224 (2012)

    Article  MathSciNet  Google Scholar 

  55. Schermer, T., Koolhof, C.: The first national Dutch Sign Language (NGT) Dictionary in book form: Van Dale Basiswoordenboek Nederlandse Gebarentaal. In: Dykstra, A., Schoonheim, T. (eds.) Proceedings of the 14th EURALEX International Congress, pp. 1555–1564. Fryske Akademy, Leeuwarden/Ljouwert, The Netherlands (2010)

    Google Scholar 

  56. Schnieder, L., Tschare, G.: Passenger information using a sign language avatar–individual travel assistance for passengers with special needs in public transport. Int. Transp. 67(1), 8–10 (2015)

    Google Scholar 

  57. Silva, I.R., Kumada, K.M.O.: Representações sobre o contexto multilíngue da surdez. Interdisciplinar–Revista de Estudos em Língua e Literatura 19, 99–114 (2013)

    Google Scholar 

  58. Silva, I.R., Nogueira, A.S., Hildebrand, H.R., Kumada, K.M.O.: O uso de jogos eletrônicos no processo ensino-aprendizagem de surdos. In: Valle, L.E.L.R., Matos, M.J.V.M., Costa, J.W. (eds.) Educação digital: a tecnologia a favor da inclusão, pp. 213–238. Penso, Porto Alegre (2013)

    Google Scholar 

  59. Siple, P.: Attentional resources and working memory: a new framework for the study of the impact of deafness on cognition. In: Emmorey, K., Lane, H. (eds.) The Signs of Language Revisited: An Anthology to Honor Ursula Bellugi and Edward Klima,  pp. 115–134. Lawrence Erlbaum Associates, London (2000)

  60. Soechting, J.F., Buneo, C.A., Herrmann, U., Flanders, M.: Moving effortlessly in three dimensions: does Donders’ law apply to arm movement? J. Neurosci. 15, 6271–6280 (1995)

    Google Scholar 

  61. Somers, H., Diaz, G.F.: Translation memory vs. example-based MT: what is the difference? Int. J. Trans. 16, 5–33 (2004)

    Google Scholar 

  62. Stein, D., Schmidt, C., Ney, H.: Analysis, preparation, and optimization of statistical sign language machine translation. Mach.Transl. 26(4), 325–357 (2012)

    Article  Google Scholar 

  63. Stokoe Jr, W.C.: Sign language structure: an outline of the visual communication system of the American deaf. Studies in Linguistics: Occasional Papers 8, pp. 1–78 (Revised 1978) (1960)

  64. Sumita, E., Iida, H., Kohyama, H.: Translating with examples: a new approach to machine translation. In: Third International Conference on Theoretical and Methodological Issues in Machine Translation of Natural Language. vol. 3, 203–212 (1990)

    Google Scholar 

  65. Suszczańska, N., Szmal, P., Francik, J.: Translating polish texts into sign language in the TGT system. In: 20th IASTED International Multi–Conference Applied Informatics AI, Innsbruck, Austria, pp. 282–287 (2002)

  66. Sze, F., Woodward, J., Tang, G., Lee, J., Cheng, K., Mak, J.: Sign language documentation in the Asia–Pacific region: a deaf-centered approach. In: Proceedings of the 5th Workshop on the Representation and Processing of Sign Languages: Interactions between Corpus and Lexicon, Language Resources and Evaluation Conference (LREC) 2012, pp. 155–158 (2012)

  67. Umeda, S., Azuma, M., Uchida, T., Miyazaki, T., Kato, N., Inoue, S., Hiruma, N.: Automatic translation system to JSL (Japanese Sign Language) about weather information. In: Proceedings of the Sign Language Translation and Avatar Technology (SLTAT 2015) (2015)

  68. Uno, Y., Kawato, M., Suzuki, R.: Formation and control of optimal trajectory in human multijoint arm movement. Biol. Cybern. 61, 89–101 (1989)

    Article  Google Scholar 

  69. Van Zijl, L., Baker, D.: South African sign language machine translation system. In: Proceedings of the 2nd International Conference on Computer Graphics, Virtual Reality, Visualisation and Interaction in Africa, Association for Computing Machinery-ACM, pp. 49–52 (2003)

  70. Veale, T., Conway, A.: Cross modal comprehension in ZARDOZ an English to sign-language translation system. In: Proceedings of the Seventh International Workshop on Natural Language Generation, Association for Computational Linguistics, pp. 249–252 (1994)

  71. Verlinden, M., Tijsseling, C., Frowein, H.: A signing Avatar on the WWW. In:  Wachsmuth, I., Sowa, T. (eds.) Gesture and Sign Language in Human–Computer Interaction. Lecture Notes in Computer Science, vol. 2298, pp. 169–172. Springer, Berlin (2002)

  72. Wittenburg, P., Brugman, H., Russel, A., Klassmann, A., Sloetjes, H.: Elan: a professional framework for multimodality research. In: Proceedings of LREC, Fifth International Conference on Language Resources and Evaluation, pp. 1556–1559 (2006)

  73. Xu, H.Y., Li, D., Tang, S.: Kinematic Inbetweening along implicit curves for motion animations. In: Proceedings of the 16th International Conference on Artificial Reality and Telexistence–Workshops, IEEE Computer Society, Washington, DC, USA, ICAT ’06, pp. 1–4 (2006)

  74. Zhao, L., Kipper, K., Schuler, W., Vogler, C., Badler, N.I., Palmer, M.: A machine translation system from English to American sign language. In: Proceedings of the 4th Conference of the Association for Machine Translation in the Americas on Envisioning Machine Translation in the Information Future, Springer-Verlag, London, UK, AMTA ’00, pp. 54–67 (2000)

Download references

Acknowledgements

This research is being supported by CNPq/MCTI-SECIS, Grant Number 458691/2013-5, and by Capes/SDH/MCTI, Grant Number 88887.091672/2014-0.

Author information

Authors and Affiliations

  1. School of Electrical and Computer Engineering, University of Campinas, Campinas, Brazil

    José Mario De Martino, Paula Dornhofer Paro Costa, Wanessa Machado do Amaral, Enzo Telles Poeta, Leandro Martin Guertzenstein Angare & Carolina Monteiro Ferreira

  2. Center for Rehabilitation Research Studies, University of Campinas, Campinas, Brazil

    Ivani Rodrigues Silva & Kate Mamhy Oliveira Kumada

  3. Institute of Language Studies, University of Campinas, Campinas, Brazil

    Carmen Zink Bolognini & Davi Faria De Conti

  4. Federal University of Alagoas, Alagoas, Brazil

    Luis Cláudius Coradine & Patrick Henrique da Silva Brito

  5. Center for Information Technology Renato Archer, Campinas, Brazil

    Ângelo Brandão Benetti

Authors
  1. José Mario De Martino
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ivani Rodrigues Silva
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Carmen Zink Bolognini
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Paula Dornhofer Paro Costa
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kate Mamhy Oliveira Kumada
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Luis Cláudius Coradine
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Patrick Henrique da Silva Brito
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Wanessa Machado do Amaral
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Ângelo Brandão Benetti
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Enzo Telles Poeta
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Leandro Martin Guertzenstein Angare
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Carolina Monteiro Ferreira
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Davi Faria De Conti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to José Mario De Martino.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

De Martino, J.M., Silva, I.R., Bolognini, C.Z. et al. Signing avatars: making education more inclusive. Univ Access Inf Soc 16, 793–808 (2017). https://doi.org/10.1007/s10209-016-0504-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10209-016-0504-x

Keywords

Search

Navigation