Advertisement

Universal Access in the Information Society

, Volume 11, Issue 2, pp 151–168 | Cite as

Hybrid paradigm for Spanish Sign Language synthesis

Long paper

Abstract

This work presents a hybrid approach to sign language synthesis. This approach allows the hand-tuning of the phonetic description of the signs, which focuses on the time aspect of the sign. Therefore, the approach retains the capacity for the performing of morpho-phonological operations, like notation-based approaches, and improves the synthetic signing performance, such as the hand-tuned animations approach. The proposed approach simplifies the input message description using a new high-level notation and storage of sign phonetic descriptions in a relational database. Such relational database allows for more flexible sign phonetic descriptions; it also allows for a description of sign timing and the synchronization between sign phonemes. The new notation, named HLSML, is a gloss-based notation focusing on message description in it. HLSML introduces several tags that allow for the modification of the signs in the message that defines dialect and mood variations, both of which are defined in the relational database, and message timing, including transition durations and pauses. A new avatar design is also proposed that simplifies the development of the synthesizer and avoids any interference with the independence of the sign language phonemes during animation. The obtained results showed an increase of the sign recognition rate compared to other approaches. This improvement was based on the active role that the sign language experts had in the description of signs, which was the result of the flexibility of the sign storage approach. The approach will simplify the description of synthesizable signed messages, thus facilitating the creation of multimedia-signed contents.

Keywords

Spanish Sign Language Sign language synthesis Graphical user interfaces Human computer interaction 

Notes

Acknowledgments

The authors would like to acknowledge the teachers of the SIGNAR academy for their participation in the evaluations and to the FPU program of the Universidad Autónoma de Madrid for its financial support. The authors are grateful to the journal’s editor and the reviewers for their comments and suggestions, which have improved this paper and proposed interesting future research directions.

References

  1. 1.
    Bangham, A., Cox, S., Elliot, R., Glauer, J., Marshal, I.: Virtual signing: capture, animation, storage and transmission—an overview of the ViSiCAST project. In: Proceedings of IEE Seminary on Speech and Language Processing for Disabled and Elderly People, pp. 1–7 (2000)Google Scholar
  2. 2.
    Battison, R.: Phonology in American Sign Language: 3-D and digit-vision. In: Proceedings of the California Linguistic Association conference, Stanford (1973)Google Scholar
  3. 3.
    Brentari, D.: Trilled movement: phonetic realization and formal representation. Lingua 98, 43–71 (1996)CrossRefGoogle Scholar
  4. 4.
    Corina, D.: Sign linguistics phonetics, phonology and morpho-syntax. Lingua 98, 73–102 (1996)CrossRefGoogle Scholar
  5. 5.
    Cox, S., Lincoln, M., Nakisa, M., Wells, M., Tutt, M., Abbott, S.: The development and evaluation of a speech-to-sign translation system to assist transactions. Int. J. Hum-comput. Interact. 16, 141–161 (2003). doi: 10.1207/S15327590IJHC1602_02 CrossRefGoogle Scholar
  6. 6.
    Elliot, R., Glauert, J., Jennings, V., Kennaway, R.: An overview of the SiGML notation and SiGML signing software system. In: Proceedings of LREC, Lisbon, pp. 98–104 (2004)Google Scholar
  7. 7.
    Elliott, R., Glauert, J., Kennaway, R., Marshal, I., Sáfár, É.: Linguistic modelling and language-processing technologies for avatar-based sign language presentation. Univ. Acess. Inf. Soc. 6, 375–391 (2008)CrossRefGoogle Scholar
  8. 8.
    Emmorey, K., Herzig, M.: Categorical versus gradient properties of classifier constructions in ASL. In: Emmorey, K. (ed.) Perspectives on Classifier Constructions in Sign Languages, pp. 221–246. Psychology Press (2003)Google Scholar
  9. 9.
    European Media Masters of Art: Vsign. http://www.vsign.nl/ (2002). Accessed 21 Feb 2011
  10. 10.
    Fotinea, S., Efthimiou, E., Caridakis, G., Karpouzis, K.: A knowledge-based sign synthesis architecture. Univ. Acess. Inf. Soc. 6, 405–418 (2008)CrossRefGoogle Scholar
  11. 11.
    Fundación CNSE: Diccionario Normativo de la Lengua de Signos Española [Spanish Sign Language Normative Dictionary]. Fundación CNSE, Madrid (2008)Google Scholar
  12. 12.
    Hanke, T.: Hamnosys—representing sign language data in language resources and language processing contexts. In: Proceedings of LREC, Lisbon, pp. 1–6 (2004)Google Scholar
  13. 13.
    Herrero, B.Á.: A practical writing of sign languages. In: Proceedings of LREC, Lisbon, pp. 37–42 (2004)Google Scholar
  14. 14.
    Herrero, B.Á.: Gramática Didáctica de la Lengua de Signos Española (LSE) [Spanish Sign Language Didactive Grammar], 1st edn. Ediciones SM, Madrid (2009)Google Scholar
  15. 15.
    Huang, Z., Elïens, A., Visser, C.: Step: a scripting language for embodied agents. In: Proceedings of Workshop on Lifelike Animated Agents, Tokyo, pp. 46–51Google Scholar
  16. 16.
    Huenerfauth, M.: Spatial representation of classifier predicates for machine translation into American Sign Language. In: Workshop on the Representation and Processing of Signed Languages. In: Proceedings of LREC, Lisbon, pp 24–31 (2004)Google Scholar
  17. 17.
    Huenerfauth, M.: Representing coordination and non-coordination in an American Sign Language. In: Proceedings of SIGACCESS, ACM, New York, pp. 44–51 (2005). doi: 10.1145/1090785.1090796
  18. 18.
    Huenerfauth, M.: A linguistically motivated model for speed and pausing in animations of American Sign Language. ACM Trans. Access. Comput. 2, 1–31 (2009). doi: 10.1145/1530064.1530067 CrossRefGoogle Scholar
  19. 19.
    Hulst, H., Mills, A.: Issues in sign linguistic: phonetics, phonology and morphosyntax. Lingua 98(1–3), 3–17 (1996)CrossRefGoogle Scholar
  20. 20.
    Hybrid Graphics: Hybrid rasteroid (2006). http://www.hybrid.fi. Accessed 20 Jan 2008
  21. 21.
    Igi, S., Ujitani, M., Tamaru, M., Yamamoto, Y., Sugita, S.: Sign-language synthesis for mobile environments. In: Proceedings of Intern Conf in Central Europe on Computer Graphics, Visualization and Computer Vision (2003)Google Scholar
  22. 22.
    Informatics and Telematics Institute: Vsigns. http://vsigns.iti.gr:8080/VSigns (2004). Accessed 21 Feb 2011
  23. 23.
    ISO/IEC 14772–1:1997: Information technology—Computer graphics and ima—Part 1: Functional specification and UTF-8 encoding. International Organization for Standardization, Geneva (1997)Google Scholar
  24. 24.
    ISO/IEC 14772–2:2004: Information technology—Computer graphics and image processing—The Virtual Reality Modeling Language (VRML)–Part 2: External authoring interface (EAI). International Organization for Standardization, Geneva (2004)Google Scholar
  25. 25.
    ISO/IEC 19774:2005: Information technology—Computer graphics and image processing—Humanoid animation (H-Anim). International Organization for Standardization, Geneva (2005)Google Scholar
  26. 26.
    Karpouzis, K., Caridakis, G., Fotinea, S.E., Efthimiou, E.: Educational resources and implementation of a Greek sign language synthesis architecture. Comput. Educ. 49, 54–74 (2007). doi: 10.1016/j.compedu.2005.06.004 CrossRefGoogle Scholar
  27. 27.
    Kennaway, R., Glauert, J., Zwitserlood, I.: Providing signed content on the internet by synthesized animation. ACM Trans. Comput. Human Interact. 14, 1–29 (2007). doi: 10.1145/1279700.1279705 CrossRefGoogle Scholar
  28. 28.
    Liddell, S.: Sources of meaning in ASL classifier predicates. In: Emmorey, K. (ed.) Perspectives on Classifier Constructions in Sign Languages, pp. 199–220. Psychology Press (2003)Google Scholar
  29. 29.
    Liddell, S., Johnson, R.: American Sign Language: the phonological base. Sign Lang. Stud. 64, 195–278 (1989)Google Scholar
  30. 30.
    López-Colino, F., Colás, J.: Providing ubiquitous access to synthetic sign language contents over multiple platforms. In: Cruz-Cunha, M., Moreira, F. (eds.) Handbook of Research on Mobility and Computing: Evolving Technologies and Ubiquitous Impacts, IGI International (In Press)Google Scholar
  31. 31.
    López-Colino, F., Garrido, J., Colás, J.: Description and synthesis of Spanish sign language classifiers. In: Proceedings of INTERACCION, AIPO, Barcelona (2009)Google Scholar
  32. 32.
    Moccozet, L., Magnenat-Thalmann, N.: Dirichelet free form deformations and their application to hand simulation. In: Proceedings of IEEE Computer Animation, pp. 93–102 (1997)Google Scholar
  33. 33.
    Muñoz Baell, I.:¿Cómo se articula la lengua de signos española? [How is the articulation of LSE?] CNSE, Madrid (1999)Google Scholar
  34. 34.
    Prillwitz, S., Leven, R., Zienert, H., Hanke, T., Herming J.: HamNoSys. Version 2.0; Hamburg Notation System for Sign Languages. An introductory guide. Verlag (1989)Google Scholar
  35. 35.
    Rocha, A., Pereira, G.: Supporting deaf sign languages in written form on the web. In: Proceedings of LREC, Lisbon, pp 26–28 (2004)Google Scholar
  36. 36.
    San Segundo, R., Barra, R., D’Haro, L.F., Montero, J.M., Cordoba, R., Ferreiros, J.: A Spanish speech to sign language translation system for assisting deaf-mute people. In: Proceedings of INTERSPEECH, Pittsburgh, USA, pp. 1399–1402 (2006)Google Scholar
  37. 37.
    San Segundo, R., Barra, R., Córdoba, R., D’Haro, L.F., Fernandez, F., Ferreiros, J., Lucas, J., Macías-Guarasa, J., Montero, J.M., Pardo, J.: Speech to sign language translation system for Spanish. Speech Commun. 50, 1009–1020 (2008). doi:  10.1016/j.specom.2008.02.001 Google Scholar
  38. 38.
    Sandler, W.: Phonological Representation of the Sign: Linearity and Nonlinearity in American sign Language. Foris Publications, Holland (1989)CrossRefGoogle Scholar
  39. 39.
    Sandler, W., Lillo-Martin, D.: Sign Language and Linguistic Universals. Cambridge University Press (2006)Google Scholar
  40. 40.
    Schembri, A.: Rethinking ‘classifiers’ in signed languages. In: Emmorey, K. (ed.) Perspectives on Classifier Constructions in Sign Languages, pp. 3–34. Psichology Press (2003)Google Scholar
  41. 41.
    Schembri, A., Jones, C., Burnham, D.: Comparing action gestures and classifier verbs of motion: evidence from Australian sign language, Taiwan sign language, and non signers’ gestures without speech. J. Deaf Stud. Deaf Edu. 10, 272–290 (2005)CrossRefGoogle Scholar
  42. 42.
    Segouat, J.: A study of sign language co articulation. In: Proceedings of SIGACCESS, ACM, New York, pp. 31–38 (2009). doi:  10.1145/1531930.1531935
  43. 43.
    Segouat, J., Braffort, A.: Toward the study of sign language co articulation: methodology proposal. In: Proceedings of the International Conference on Advances in Computer-Human Interaction, IEEE Computer Society, Los Alamitos, CA, USA, pp. 369–374 (2009). doi: 10.1109/ACHI.2009.25
  44. 44.
    Solina, F., Krapez, S., Jaklic, A., Komac, V.: Multimedia dictionary and synthesis of sign language. In: Rahman, I. (ed.) Design and Management of Multimedia Information Systems, pp. 268–281. Idea Group Publishing (2001)Google Scholar
  45. 45.
    Stokoe, W.: Sign language structure: an outline of the visual communication systems of the American deaf. Stud. Linguist. Occas. Papers 8, 1–78 (1960)Google Scholar
  46. 46.
    Stokoe, W.: Sign Language Structure: the First Linguistic Analysis of American Sign Language. Linstok Press Incorporated (1978)Google Scholar
  47. 47.
    Supalla, T.: Revisiting visual analogy in ASL classifier predicates In: Emmorey, K. (ed.) Perspectives on Classifier Constructions in Sign Languages, pp. 249–257. Psychology Press (2003)Google Scholar
  48. 48.
    Sutton, V.: Signwriting. http://www.signwriting.org/ (1974). Accessed 21 Feb 2011
  49. 49.
    VCom3D.: Signsmith studio. http://www.vcom3d.com/signsmith.php (2009). Accessed 21 Feb 2011
  50. 50.
    Zhao, L., Kipper, K., Schuler, W., Vogler, C., Badler, N.I., Palmer, M.: A machine translation system from English to American Sign Language. In: Proceedings of AMTA, Springer, London, UK, pp. 54–67 (2000)Google Scholar
  51. 51.
    Zijl, L.: South African sign language machine translation project. In: Proceedings of SIGACCESS, ACM, New York, NY, USA, pp. 233–234 (2006). doi: 10.1145/1168987.1169031
  52. 52.
    Zwiterslood, I., Verlinden, M., Ros, J., Schoot, S.: Synthetic signing for the deaf: eSign. In: Proceedings of CVHI, Granada, Spain (2004)Google Scholar

Copyright information

© Springer-Verlag 2011

Authors and Affiliations

  1. 1.Human Computer Technology LaboratoryUniversidad Autónoma de MadridCantoblancoSpain

Personalised recommendations