Skip to main content
SpringerLink
Log in

An Approach to Sri Lankan Sign Language Recognition Using Deep Learning with MediaPipe

  • Conference paper
  • First Online:
Digital Technologies and Applications (ICDTA 2022)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 454))

Included in the following conference series:

  • 985 Accesses

Abstract

Currently, there are millions of people around the world with speech and hearing impairments, and according to the latest census, there are nearly 70,000 people who use Sri Lankan Sign Language. Sign language is a visual language, and it is the main medium of communication in their daily conversations. But they face obstacles when communicating with people who do not know sign language. There are communication barriers in different contexts, such as in work environments, knowledge exchange, and message sharing. Therefore, technology should play a major role in helping people with these hearing and speech impairments to improve their quality of life. This research aims to suggest models that use Google MediaPipe Hand Pose landmarks to identify Sri Lankan Sign Language. Moreover, this article compares vision-based approaches with convolutional neural network (CNN) and recurrent neural network (RNN). We also considered activation functions (such as ReLU, Linear, and Softmax), loss functions (mean squared error (MSE) and Categorical_crossentropy), and optimizations (Adam and Stochastic Gradient Descent (SGD)). The result showed that most algorithms built with Long Short-Term Memory (LSTM), CNN, and CNN-LSTM achieved an accuracy greater than 95%, both with the training dataset and the test dataset. In particular, models with MSE as the loss function and Adam as the optimizer showed higher accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 249.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Deafness and hearing loss. http://www.who.int/mediacentre/factsheets/fs300/en/. Accessed 01 Oct 2021

  2. Fernando, P., Wimalaratne, P.: Sign language translation approach to SinhaleseLanguage. GSTF J. Comput. (JoC) 5(1), 1–9 (2016). https://doi.org/10.7603/s40601-016-0009-8

    Article  Google Scholar 

  3. Nirosha, W., Choolangika, S., Vinodya, S.: Sign language translator for deaf and speech impaired people using convolutional neural network. In: 12th International Research Conference on General Sir John Kotelawala Defence (2019)

    Google Scholar 

  4. Malinda, P., Gayan, M.: Computer interpreter for translating written Sinhala to Sinhala sign language. In: OUSL J. 12(1), 70–90 (2017). https://doi.org/10.4038/ouslj.v12i1.7377

  5. Bach, K., Duong, P., Ha, P., Anh, B., Son, N.: Vietnamese sign language detection using Mediapipe. In: 10th International Conference on Software and Computer Applications (ICSCA 2021) (2021). https://doi.org/10.1145/3457784.3457810

  6. Ashok, S., Kiran, R.: Vision based Indian sign language character recognition. J. Theor. Appl. Inf. Technol. 67(3) (2014)

    Google Scholar 

  7. Kishore, P., Panakala, K.: A video based Indian Sign Language Recognition System (INSLR) using wavelet transform and fuzzy logic. Int. J. Eng. Technol. 4(5) (2012). https://doi.org/10.7763/IJET.2012.V4.427

  8. Deepak, S., Deepak, V., Poras, K.: LabVIEW based sign language trainer cum portable display unit for the speech impaired. In: 2015 Annual IEEE India Conference (INDICON) (2015). https://doi.org/10.1109/INDICON.2015.7443381

  9. Sako, S., Hatano, M., Kitamura, T.: Real-time Japanese sign language recognition based on three phonological elements of sign. In: Stephanidis, C. (ed.) HCI 2016. CCIS, vol. 618, pp. 130–136. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-40542-1_21

    Chapter  Google Scholar 

  10. Madushanka, A., Senevirathne, R., Wijesekara, L., Arunatilake, S., Sandaruwan, K.: Framework for Sinhala sign language recognition and translation using a wearable armband. In: Sixteenth International Conference on Advances in ICT for Emerging Regions (2016). https://doi.org/10.1109/ICTER.2016.7829898

  11. Griselda, S., Jorge, C., Mario, B., Apolonio, P.: Recognition and classification of sign language for Spanish. Computacion y Sistemas22(1), 271–277 (2018). https://doi.org/10.13053/cys-22-1-2780

  12. Hardik, R., Vishal, D., Dhroov, B., Hema, N.: Automated sign language interpreter. In: Eleventh International Conference on Contemporary Computing (2018). https://doi.org/10.1109/IC3.2018.8530658

  13. Ponlawat, C., Kosin, C.: Backhand-view-based continuous-signed-letter recognition using a rewound video sequence and the previous signed-letter information. IEEE Access 9, 40187–40197 (2021). https://doi.org/10.1109/ACCESS.2021.3063203

  14. Tharwat, A., Gaber, T., Hassanien, A.E., Shahin, M.K., Refaat, B.: SIFT-based arabic sign language recognition system. In: Abraham, A., Krömer, P., Snasel, V. (eds.) Afro-European Conference for Industrial Advancement. AISC, vol. 334, pp. 359–370. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-13572-4_30

    Chapter  Google Scholar 

  15. Zuzanna, P., Carlos-D, M.: Sign language gesture classification using neural networks. In: IBER SPEECH (2018). https://doi.org/10.21437/IberSPEECH.2018-27

  16. Aulia, P., Erdefi, R., Dadan, H.: Human skeleton feature extraction from 2-dimensional video of Indonesian language sign system (SIBI [Sistem Isyarat Bahasa Indonesia]) gestures. In: 5th International Conference on Computing and Artificial Intelligence, pp. 100–105 (2019). https://doi.org/10.1145/3330482.3330484

  17. Sobia F., Yasar, A.: CNN and traditional classifiers performance for sign language recognition. In: 3rd International Conference on Machine Learning and Soft Computing, pp. 192–196 (2019). https://doi.org/10.1145/3310986.3311011

  18. Darmana, A., Erdefi, R.: Generating of Sign System for Bahasa Indonesia (SIBI) root word gestures using deep temporal sigmoid belief network. In: ICCAI 2019: Proceedings of the 2019 5th International Conference on Computing and Artificial Intelligence, pp. 221–225 (2019). https://doi.org/10.1145/3330482.3330494

  19. Pisit, N., Patcharee, M., Tatpong, K.: Thai finger spelling localization and classification under complex background using a YOLO-based deep learning. In: 11th International Conference on Computer Modeling and Simulation, pp. 230–233 (2019). https://doi.org/10.1145/3307363.3307403

  20. Huang, C., Wang, F., Zhang, R.: Sign language recognition based on CBAM-ResNet. In: International Conference on Artificial Intelligence and Advanced Manufacturing, no. 48, pp. 1–6 (2019). https://doi.org/10.1145/3358331.3358379

  21. Kirtee, P., Sreemathy, R., Akshay, V.: Recognition of Indian sign language alphabets for hearing and speech impaired people using deep learning. SSRN Electron. J. (2019). https://doi.org/10.2139/ssrn.3430055

  22. Heike, B., Felix, L., Kazuhiro, N., Yuji, N.: Learning three-dimensional skeleton data from sign language video. ACM Trans. Intell. Syst. Technol. 11(3) (2020). https://doi.org/10.1145/3377552

  23. Neha, N., Saloni, M., Sandhya, A., Ekansh, T.: A dynamic gesture recognition system for mute person. In: Goyal, D., Gupta, A.K., Piuri, V., Ganzha, M., Paprzycki, M. (eds.) Proceedings of the Second International Conference on Information Management and Machine Intelligence. LNNS, vol. 166, pp. 33–39. Springer, Singapore (2020). https://doi.org/10.1007/978-981-15-9689-6_4

  24. Amrita, T., Pujan, B., Sarmila, U., Shirish, S., Subarna, S.: Real time sign language recognition and speech generation. J. Innov. Image Process. 2(2), 65–76 (2020). https://doi.org/10.36548/jiip.2020.2.001

  25. Taniya, S., Soumi, P., Subhadip, B., Ayatullah, M.: Hand sign recognition from depth images with multi-scale density features for deaf mute persons. International Conference on Computational Intelligence and Data Science (2019). https://doi.org/10.1016/j.procs.2020.03.243

  26. Biao, X., Shiliang, H., Zhongfu, Y.: Application of tensor train decomposition in S2VT model for sign language recognition. IEEE Access 9, 35646–35653 (2021). https://doi.org/10.1109/ACCESS.2021.3059660

  27. Astri, N., Fairuz, A.: Sign language recognition using principal component analysis and support vector machine. Int. J. Appl. Inf. Technol. 4(1) (2021). https://doi.org/10.25124/ijait.v4i01.3015

  28. Chengcheng, W., Jian, Z., Wengang, Z., Houqiang, Li.: Semantic boundary detection with reinforcement learning for continuous sign language recognition. In: IEEE Trans. Circuits Syst. Video Technol. 31, 1138–1149 (2021). https://doi.org/10.1109/TCSVT.2020.2999384

  29. Mohamed, B., et al.: Arabic Sign language recognition system using 2D hands and body skeleton data. IEEE Access 9, 59612–59627 (2021). https://doi.org/10.1109/ACCESS.2021.3069714

  30. Lu, J., Nguyen, M., Yan, W.Q.: Sign language recognition from digital videos using deep learning methods. In: Nguyen, M., Yan, W.Q., Ho, H. (eds.) ISGV 2021. CCIS, vol. 1386, pp. 108–118. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-72073-5_9

    Chapter  Google Scholar 

  31. Prikhodko, A., Grif, M., Bakaev, M.: Sign language recognition based on notations and neural networks. In: Alexandrov, D.A., Boukhanovsky, A.V., Chugunov, A.V., Kabanov, Y., Koltsova, O., Musabirov, I. (eds.) DTGS 2020. CCIS, vol. 1242, pp. 463–478. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-65218-0_34

    Chapter  Google Scholar 

  32. Anusorn, C., Kritsana, S., Thidalak, Y.: Thai sign language recognition: an application of deep neural network. In: 2021 Joint International Conference on Digital Arts, Media and Technology with ECTI Northern Section Conference on Electrical, Electronics, Computer and Telecommunication Engineering (2021). https://doi.org/10.1109/ECTIDAMTNCON51128.2021.9425711

Download references

Acknowledgement

I express my great appreciation to the Department of Computing and Information Systems, Faculty of Applied Sciences, Sabaragamuwa University of Sri Lanka for providing opportunities, guidance, advice, tremendous encouragement, endless experimental recommendations, a lot of precious time, and endless assistance for the success of this study.

Author information

Authors and Affiliations

  1. Department of Computing and Information Systems, Sabaragamuwa University of Sri Lanka, Belihuloya, Sri Lanka

    Randika Jeewantha Herath & Piumi Ishanka

Authors
  1. Randika Jeewantha Herath
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Piumi Ishanka
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Randika Jeewantha Herath .

Editor information

Editors and Affiliations

  1. Ecole Nationale des Sciences Appliquées, Fez, Morocco

    Saad Motahhir

  2. Faculty of Sciences, Sidi Mohamed Ben Abdellah University, Fez, Morocco

    Badre Bossoufi

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Herath, R.J., Ishanka, P. (2022). An Approach to Sri Lankan Sign Language Recognition Using Deep Learning with MediaPipe. In: Motahhir, S., Bossoufi, B. (eds) Digital Technologies and Applications. ICDTA 2022. Lecture Notes in Networks and Systems, vol 454. Springer, Cham. https://doi.org/10.1007/978-3-031-01942-5_45

Download citation

Publish with us

Policies and ethics

Search

Navigation