Skip to main content
SpringerLink
Log in

Sign language detection using convolutional neural network

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

Sign language recognition is an important social issue to be addressed which can benefit the deaf and hard of hearing community by providing easier and faster communication. Some previous studies on sign language recognition have used complex input modalities and feature extraction methods, limiting their practical applicability. This research aims to compare two custom-made convolutional neural network (CNN) models for recognizing American Sign Language (ASL) letters from A to Z, and determine which model performs better. The proposed models utilize a combination of CNN and Softmax activation function, which are powerful and widely used classification methods in the field of computer vision. The purpose of the proposed study is to compare the performance of two specially created CNN models for identifying 26 distinct hand signals that represent the 26 English alphabets. The study found that Model_2 had better overall performance than Model_1, with an accuracy of 98.44% and F1 score 98.41%. However, the performance of each model varied depending on the specific label, suggesting that the choice of model may depend on the specific use case and the labels of interest. This research contributes to the growing field of sign language recognition using deep learning techniques and highlights the importance of designing custom models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29
Fig. 30

Similar content being viewed by others

Data availability

The datasets used/generated and/or analysed during the current study are available from the corresponding author on reasonable request.

References

  • Abraham A, Rohini V (2018) Real time conversion of sign language to speech and prediction of gestures using Artificial Neural Network. Proc Comput Sci 143:587–594

    Article  Google Scholar 

  • Adaloglou N, Chatzis T, Papastratis I, Stergioulas A, Papadopoulos GT, Zacharopoulou V, Daras P (2021) A comprehensive study on deep learning-based methods for sign language recognition. IEEE Trans Multimed 24:1750–1762

    Article  Google Scholar 

  • Adeyanju IA, Bello OO, Adegboye MA (2021) Machine learning methods for sign language recognition: a critical review and analysis. Intell Syst Appl 12:200056

    Google Scholar 

  • Adithya V, Rajesh R (2020) A deep convolutional neural network approach for static hand gesture recognition. Proc Comput Sci 171:2353–2361

    Article  Google Scholar 

  • Al-Shamayleh AS, Ahmad R, Abushariah MA, Alam KA, Jomhari N (2018) A systematic literature review on vision based gesture recognition techniques. Multimed Tools Appl 77:28121–28184

    Article  Google Scholar 

  • Amrutha K, Prabu P (2021) ML based sign language recognition system. In: 2021 International Conference on innovative trends in information technology (ICITIIT), pp 1–6. IEEE

  • Bamwenda J, Özerdem MS (2019) Static hand gesture recognition system using artificial neural networks and support vector machine. Dicle Üniversitesi Mühendislik Fakültesi Mühendislik Dergisi 10(2):561–568

    Google Scholar 

  • Cassim MR, Parry J, Pantanowitz A, Rubin DM (2022) Design and construction of a cost-effective, portable sign language to speech translator. Inform Med Unlock 30:100927

    Article  Google Scholar 

  • Davey S, Davey A, Jain R (2020) Impact of Community Oriented Ear Care (COEC) on national programme for control of deafness in India: a critical look. Adv Treat ENT Disord 4(1):001–002

    Article  Google Scholar 

  • Dima TF, Ahmed ME (2021) Using YOLOv5 algorithm to detect and recognize American sign language. In: 2021 International Conference on information technology (ICIT), pp 603–607. IEEE

  • Gilorkar NK, Ingle MM (2014) Real time detection and recognition of Indian and American sign language using sift. Int J Electron Commun Eng Technol 5(5):11–18

    Google Scholar 

  • Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp 580–587

  • https://www.kaggle.com/. Accessed 24 Dec 2022

  • Indolia S, Goswami AK, Mishra SP, Asopa P (2018) Conceptual understanding of convolutional neural network-a deep learning approach. Proc Comput Sci 132:679–688

    Article  Google Scholar 

  • Jenkins J, Rashad S (2022) LeapASL: A platform for design and implementation of real time algorithms for translation of American Sign Language using personal supervised machine learning models. Softw Impacts 12:100302

    Article  Google Scholar 

  • Katoch S, Singh V, Tiwary US (2022) Indian Sign Language recognition system using SURF with SVM and CNN. Array 14:100141

    Article  Google Scholar 

  • Kaur K, Kumar P (2016) HamNoSys to SiGML conversion system for sign language automation. Proc Comput Sci 89:794–803

    Article  Google Scholar 

  • Kumar VK, Goudar RH, Desai VT (2015) Sign language unification: the need for next generation deaf education. Proc Comput Sci 48:673–678

    Article  Google Scholar 

  • Kuznetsova A, Leal-Taixé L, Rosenhahn B (2013) Real-time sign language recognition using a consumer depth camera. In: Proceedings of the IEEE International Conference on computer vision workshops, pp 83–90

  • Li Z, Nie F, Chang X, Yang Yi, Zhang C, Sebe N (2018a) Dynamic affinity graph construction for spectral clustering using multiple features. IEEE Trans Neural Netw Learn Syst 29(12):6323–6332

    Article  MathSciNet  Google Scholar 

  • Li Z, Nie F, Chang X, Nie L, Zhang H, Yang Yi (2018b) Rank-constrained spectral clustering with flexible embedding. IEEE Trans Neural Netw Learn Syst 29(12):6073–6082

    Article  MathSciNet  Google Scholar 

  • Li Z, Xu P, Chang X, Yang L, Zhang Y, Yao L, Chen X (2023) When object detection meets knowledge distillation: A survey. IEEE Trans Pattern Anal Mach Intell 45:10555–10579

    Article  Google Scholar 

  • Liu Y, Liu X, Liu X (2020) Real-time hand gesture recognition using motion history images. In: 2020 International Conference on image and vision computing New Zealand (IVCNZ), 2020, pp 1–6

  • Luqman H (2022) An efficient two-stream network for isolated sign language recognition using accumulative video motion. IEEE Access 10:93785–93798

    Article  Google Scholar 

  • Pigou L, Dieleman S, Kindermans PJ, Schrauwen B (2015) Sign language recognition using convolutional neural networks. In: Computer Vision-ECCV 2014 Workshops: Zurich, Switzerland, September 6–7 and 12, 2014, Proceedings, Part I 13, pp 572–578. Springer International Publishing

  • Rajaganapathy S, Aravind B, Keerthana B, Sivagami M (2015) Conversation of sign language to speech with human gestures. Proc Comput Sci 50:10–15

    Article  Google Scholar 

  • Sarma D, Bhuyan MK (2021) Methods, databases and recent advancement of vision-based hand gesture recognition for hci systems: a review. SN Comput Sci 2(6):436

    Article  Google Scholar 

  • Sethi A, Hemanth S, Kumar K, Bhaskara Rao N, Krishnan R (2012) Signpro—an application suite for deaf and dumb. IJCSET 2(5):1203–1206

    Google Scholar 

  • Sharma P, Anand RS (2021) A comprehensive evaluation of deep models and optimizers for Indian sign language recognition. Graph vis Comput 5:200032

    Article  Google Scholar 

  • Starner T, Pentland A (1997) Real-time american sign language recognition from video using hidden Markov models. Motion-based Recognit, pp 227–243

  • Sunitha KA, Saraswathi PA, Aarthi M, Jayapriya K, Lingam S (2016) Deaf mute communication interpreter-a review. Int J Appl Eng Res 11:290–296

    Google Scholar 

  • Tan M, Pang R, Le QV (2020).Efficientdet: scalable and efficient object detection. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 10781–10790

  • Vogler C, Metaxas D (2004) Handshapes and movements: multiple-channel ASL recognition Lecture Notes in Artificial Intelligence Springer. In: Lecture Notes in Artificial Intelligence, 2915, 247–258

  • Wang X, Zhang L, Zhang J (2019) “Hand Gesture Recognition using Temporal Convolutional Networks,” in. International Conference on Robotics and Automation (ICRA) 2019:1–6

    Google Scholar 

  • Yan C, Chang X, Li Z, Guan W, Ge Z, Zhu L, Zheng Q (2021) Zeronas: differentiable generative adversarial networks search for zero-shot learning. IEEE Trans Pattern Anal Mach Intell 44(12):9733–9740

    Article  Google Scholar 

  • Zhang L, Chang X, Liu J, Luo M, Li Z, Yao L, Hauptmann A (2022) Tn-zstad: Transferable network for zero-shot temporal activity detection. IEEE Trans Pattern Anal Mach Intell 45(3):3848–3861

    Google Scholar 

  • Zhou R, Chang X, Shi L, Shen Y-D, Yang Yi, Nie F (2019) Person reidentification via multi-feature fusion with adaptive graph learning. IEEE Trans Neural Netw Learn Syst 31(5):1592–1601

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Science & Engineering Department, JIS College of Engineering, Kalyani, West Bengal, India

    Pranati Rakshit, Sarbajeet Paul & Shruti Dey

Authors
  1. Pranati Rakshit
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sarbajeet Paul
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shruti Dey
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pranati Rakshit.

Ethics declarations

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Ethical statement

We declare that there are no ethical issues for human or animal rights in the work presented here.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rakshit, P., Paul, S. & Dey, S. Sign language detection using convolutional neural network. J Ambient Intell Human Comput 15, 2399–2424 (2024). https://doi.org/10.1007/s12652-024-04761-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-024-04761-7

Keywords

Search

Navigation