Abstract
This paper describes our Czech sign language synthesis that converts a Czech text into a series of skeletal poses. Our main goal is to avoid demanding handcrafted annotations of videos and to avoid a manual mapping between sign language glosses and skeletal poses. Thus, instead of solving these task separately, we join a model of an implicit neural-network-based translator and a model of the mapping between sign language glosses and we train both models together. For this purpose, we propose a simple differentiable operation that decomposes input symbols and it allows to produce a required series without any recurrent mechanism. We used The OpenPose toolbox to automatically extract skeletal poses and we designed a gradient-descend-based algorithm that converts a 2D skeleton model to a 3D skeleton model in order to fix misplaced and missing joints. Weather forecast parts of The daily news in Czech sign language were used to obtain our training and testing data. Our experiments demonstrate the benefit of the implicit translator and an ability of the designed sign language synthesis system to produce naturally formed skeletal poses.
This work was supported by the European Regional Development Fund under the project AI&Reasoning (reg. no. CZ.02.1.01/0.0/0.0/15 003/0000466). Access to computing and storage facilities owned by parties and projects contributing to the National Grid Infrastructure MetaCentrum provided under the programme “Projects of Large Research, Development, and Innovations Infrastructures” (CESNET LM2015042), is greatly appreciated.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Almeida, I., Coheur, L., Candeias, S.: Coupling natural language processing and animation synthesis in Portuguese sign language translation. In: Proceedings of the Fourth Workshop on Vision and Language, pp. 94–103. Association for Computational Linguistics, Lisbon (2015)
Camgoz, N.C., Hadfield, S., Koller, O., Ney, H., Bowden, R.: Neural sign language translation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
Cuturi, M., Blondel, M.: Soft-DTW: a differentiable loss function for time-series. In: ICML (2017)
Dey, R., Salemt, F.M.: Gate-variants of gated recurrent unit (GRU) neural networks. In: 2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS), pp. 1597–1600 (2017)
Jahangiri, E., Yuille, A.L.: Generating multiple diverse hypotheses for human 3D pose consistent with 2D joint detections. In: 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), pp. 805–814 (2017)
Joo, H., Simon, T., Sheikh, Y.: Total capture: a 3d deformation model for tracking faces, hands, and bodies. CoRR abs/1801.01615 (2018)
Kanis, J., Müller, L.: Advances in Czech - signed speech translation. In: 12th International Conference on Text, Speech and Dialogue, September 2009, pp. 48–55 (2009)
Kanis, J., Zahradil, J., Jurčíček, F., Müller, L.: Czech-sign speech corpus for semantic based machine translation. In: 9th International Conference on Text, Speech and Dialogue, Brno, September 2006, pp. 613–620 (2006)
Krňoul, Z., Kanis, J., Železný, M., Müller, L.: Czech text-to-sign speech synthesizer. In: 4th International Workshop on Machine Learning for Multimodal Interaction, Brno, June 2007, pp. 180–191 (2008)
Krňoul, Z., Železný, M., Müller, L., Kanis, J.: Training of coarticulation models using dominance functions and visual unit selection methods for audio-visual speech synthesis. In: 9th International Conference on Spoken Language Processing/INTERSPEECH 2006, Pittsburgh, PA, pp. 585–588. International Speech and Communication Association (2006)
Naert, L., Larboulette, C., Gibet, S.: Coarticulation analysis for sign language synthesis. In: Antona, M., Stephanidis, C. (eds.) Universal Access in Human-Computer Interaction. Designing Novel Interactions. pp. 55–75, Springer, Cham (2017)
Stoll, S., Camgöz, N.C., Hadfield, S., Bowden, R.: Sign language production using neural machine translation and generative adversarial networks. In: BMVC (2018)
Vaswani, A., et al.: Attention is all you need. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems 30, pp. 5998–6008. Curran Associates, Inc. (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Zelinka, J., Kanis, J., Salajka, P. (2019). NN-Based Czech Sign Language Synthesis. In: Salah, A., Karpov, A., Potapova, R. (eds) Speech and Computer. SPECOM 2019. Lecture Notes in Computer Science(), vol 11658. Springer, Cham. https://doi.org/10.1007/978-3-030-26061-3_57
Download citation
DOI: https://doi.org/10.1007/978-3-030-26061-3_57
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-26060-6
Online ISBN: 978-3-030-26061-3
eBook Packages: Computer ScienceComputer Science (R0)