Abstract
In this paper, we developed HearUsNow, a deep learning-based system designed to assist deaf teenagers in correcting speech order and learning sign language, thereby improving their communication skills. This interactive system incorporates a deep learning-based back-end with a user-friendly front-end interface, aiming to bridge the gap between these individuals and their real-world communication needs. In this system, deaf teenagers receive corrections on sentence structure and accompanying sign language instructional images after entering sentences, and sign language practice is facilitated through a camera interface that provides real-time feedback. The results of a pilot study (n = 6) showed high usability, interaction fluency, and accuracy of correction and recognition of the interactive system. Over a seven-day period, the participants mastered more than 30 gesture combinations for everyday communication, and parents noted improved self-confidence and parent-child relationships.
W. Guo and J. Bai—Co-first authors of this poster.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Aran, O., et al.: Signtutor: an interactive system for sign language tutoring. IEEE Multimedia 16, 81–93 (2009)
Hein, Z., Htoo, T.P., Aye, B., Htet, S.M., Ye, K.Z.: Leap motion based myanmar sign language recognition using machine learning. In: 2021 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (ElConRus), pp. 2304–2310. IEEE (2021)
Rastogi, R., Mittal, S., Agarwal, S.: A novel approach for communication among blind, deaf and dumb people. In: 2015 2nd International Conference on Computing for Sustainable Global Development (INDIACom), pp. 605–610. IEEE (2015)
Marschark, M., Spencer, P.E.: The Oxford Handbook of Deaf Studies, Language, and Education, vol. 2. Oxford University Press, Oxford (2010)
Sultan, A., Makram, W., Kayed, M., Ali, A.A.: Sign language identification and recognition: a comparative study. Open Comput. Sci. 12, 191–210 (2022)
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. (2018)
Cui, Y., Che, W., Liu, T., Qin, B., Wang, S., Hu, G.: Revisiting pre-trained models for Chinese natural language processing. arXiv preprint arXiv:2004.13922. (2020)
Rastgoo, R., Kiani, K., Escalera, S.: Sign language recognition: a deep survey. Expert Syst. Appl. 164, 113794 (2021)
Guo, W., Li, S., Zhang, Z., Chen, Z., Chang, K.H., Wang, S.: A “magic world” for children: design and development of a serious game to improve spatial ability. Comput. Anim. Virtual Worlds 34(3), e2181 (2023)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Guo, W., Bai, J., Li, H., Chang, K.H., Xu, J. (2024). Design and Development of a Deep Learning-Based Sign Language Learning Aid for Deaf Teenagers. In: Stephanidis, C., Antona, M., Ntoa, S., Salvendy, G. (eds) HCI International 2023 – Late Breaking Posters. HCII 2023. Communications in Computer and Information Science, vol 1957. Springer, Cham. https://doi.org/10.1007/978-3-031-49212-9_20
Download citation
DOI: https://doi.org/10.1007/978-3-031-49212-9_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-49211-2
Online ISBN: 978-3-031-49212-9
eBook Packages: Computer ScienceComputer Science (R0)