Skip to main content
SpringerLink
Log in

Automated Indian sign language recognition system by fusing deep and handcrafted feature

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The deaf community faces some major challenges due to the communication gap with the hearing community. The traditional approach of employing a Sign Language (SL) interpreter is not an efficient and cost-effective solution to this problem. Thus, an automated Sign Language Recognition System (SLRS) is needed to provide an efficient and reliable solution. Existing SLRS for dynamic SL recognition utilizes the CNN-LSTM architecture, which has accomplished satisfactory results. However, spatial features extracted through Convolutional Neural Network (CNN) are insufficient for recognizing SL word that consists of identical hand orientation and multiple viewing angles. Thus, this paper proposes an SLRS named Automated Indian Sign Language Recognition System for Emergency Words (AISLRSEW) for recognizing ISL words which are frequently used in an emergency situation. The proposed AISLRSEW uses a combination of CNN and local handcrafted features to resolve the issue of identifying SL words with identical hand orientation and multiple viewing angles, which improves the recognition accuracy. The performance of the proposed AISLRSEW is evaluated with two fold cross-validation method and compared with existing models. The proposed model has achieved an average accuracy of 94.42%, which is comparatively better than existing models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability

The authors declare that all data supporting the findings of this study are available within the article.

References

  1. Adithya V, Reghunadhan R (2021) "Applying deep neural networks for the automatic recognition of sign language words: A communication aid to deaf agriculturists." Expert Syst Appl, pp-1-12

  2. Aditya V, Rajesh R (2020) Hand gestures for emergency situations: A video dataset based on words from Indian sign language.Data in Brief 31. https://doi.org/10.1016/j.dib.2020.106016

  3. Aly S, Aly W (2020) Deep ArSLR: A Novel Signer-Independent Deep Learning Framework for Isolated Arabic Sign Language Gestures Recognition. IEEE Access 8:83199–83212

    Article  Google Scholar 

  4. Ansari MA, Singh DK (2019) An approach for human machine interaction using dynamic hand gesture recognition. In: 2019 IEEE Conference on Information and Communication Technology, pp 1–6. https://doi.org/10.1109/CICT48419.2019.9066173

  5. Aparna C, Geetha M (2020) CNN and stacked LSTM model for Indian sign language recognition. Machine Learning and Metaheuristics Algorithms, and Applications, 126–134. https://doi.org/10.1007/978-981-15-4301-2_10

  6. Athira PK, Sruthi CJ, Lijiya A (2019) Signer independent sign language recognition with co-articulation elimination from live videos: an indian scenario.Journal of King Saud University - Computer and Information Sciences 34(3):771–781.https://doi.org/10.1016/j.jksuci.2019.05.002

  7. Bhatti UA, Huang M, Wang H, Zhang Y, Mehmood A, Di W (2018) Recommendation system for immunization coverage and monitoring. Hum Vaccin Immunother 14(1):165–171. https://doi.org/10.1080/21645515.2017.1379639

  8. Bhatti UA, Huang M, Zhang DWY, Mehmood A, Han H (2019) Recommendation system using feature extraction and pattern recognition in clinical care systems. Enterp Inf Syst 13(3):329–351

    Article  Google Scholar 

  9. Bhatti UA, Yu Z, Chanussot J, Zeeshan Z, Yuan L, Luo W, Nawaz SA, Bhatti MA, Ain QU, Mehmood A (2021) Local Similarity-Based Spatial–Spectral Fusion Hyperspectral Image Classification with Deep CNN and Gabor Filtering. IEEE Trans Geosci Remote Sens Volume: 60:1–15

    Article  Google Scholar 

  10. Bhatti UA, Zeeshan Z, Nizamani MM, Bazai S, Yu Z, Yuan L (2022) Assessing the change of ambient air quality patterns in Jiangsu Province of China pre-to post-COVID-19. Chemosphere 288(Pt 2):132569. https://doi.org/10.1016/j.chemosphere.2021.132569

  11. Breland SD, Skriubakken SB, Dayal A, Jha A, Yalavarthy PK (2021) Deep Learning-Based Sign Language Digits Recognition from Thermal Images with Edge Computing System. IEEE Sens J 21(9):10445–10453

    Article  Google Scholar 

  12. Cai Y et al (2021) YOLOv4-5D: An Effective and Efficient Object Detector for Autonomous Driving. IEEE Trans Instrum Meas 70:1–13. https://doi.org/10.1109/TIM.2021.3065438

  13. Chowdhary CL, Patel PV, Kathrotia KJ, Perumal MAK, Ijaz MF (2020) Analytical study of hybrid techniques for image encryption and decryption. Sensors 20(18):5162

    Article  Google Scholar 

  14. Das S, Biswas SK, Chakaraborty M, Purkayastha B (2022) “A Review on Sign Language Recognition (SLR) System: ML and DL for SLR”, IEEE Int Conf Intell Syst, Smart Green Technol (ICISSGT), pp. 177–182

  15. Das S Das S, Biswas SK, Chakaraborty M, Purkayastha B (2022) “Intelligent Indian Sign Language Recognition Systems: A Critical Review”, ICT Syst Sustain, pp. 703–713

  16. Dhingra, N, Kunz, A (2019) Res3atn - deep 3D residual attention network for hand gesture recognition in videos. In 2019 international conference on 3D vision (3DV) (pp. 491–501)

  17. Dutta KK, Bellary S (2017) “Machine Learning Techniques for Indian Sign Language Recognition”, Int Conf Current Trends Comput Electr Electron Commun (ICCTCEEC), pp. 333–336

  18. Fang Y, Liu J, Li J, Cheng J, Hu J, Yi D, Xiao X, Bhatti UA (2022) Robust zero-watermarking algorithm for medical images based on SIFT and Bandelet-DCT. Multimed Tools Appl 81(12):16863–16879

    Article  Google Scholar 

  19. Gupta B, Shukla P, Mittal A (2016) “K-nearest correlated neighbor classification for Indian sign language gesture recognition using feature fusion”, in International Conference on Computer Communication and Informatics (ICCCI)

  20. Hoang, NN, Lee, G-S, Kim, S-H, Yang, H-J (2018) A real-time multimodal hand gesture recognition via 3D convolutional neural network and key frame extraction. In proceedings of the 2018 international conference on machine learning and machine intelligence (pp. 32–37)

  21. Hore S, Chatterjee S, Santhi V , Dey N, Ashour AS, Balas V, Shi F (2017) “Indian Sign Language Recognition Using Optimized Neural Networks”, Inf Technol Intell Transport Syst, pp. 553–563

  22. Hussain R, Karbhari Y, Ijaz MF, Woźniak M, Singh PK, Sarkar R (2021) Revise-net: exploiting reverse attention mechanism for salient object detection. Remote Sens 13(23):4941

    Article  Google Scholar 

  23. Ismail MH, Dawwd SA, Ali FH (2022) Dynamic hand gesture recognition of Arabic sign language by using deep convolutional neural networks. Ind J Electr Eng Comput Sci 25(2):952–962

    Google Scholar 

  24. Janani T, Ramanan A (2017) Feature Fusion for Efficient Object Classification Using Deep and Shallow Learning. Int J Mach Learn Comput 7(5):123–127

    Article  Google Scholar 

  25. Jayadeep G, Vishnupriya NV, Venugopal V, Vishnu S, Geetha M (2020) Mudra: convolutional neural network based indian sign language translator for banks. In: 2020 4th International Conference on Intelligent Computing and Control Systems (ICICCS), pp 1228–1232. https://doi.org/10.1109/ICICCS48265.2020.9121144

  26. Kaur P, Kumar Y, Ahmed S, Alhumam A, Singla R et al (2022) Automatic license plate recognition system for vehicles using a cnn. Comput Mater Contin 71(1):35–50. https://doi.org/10.32604/cmc.2022.017681

  27. Tareen SAK, Saleem Z (2018) A comparative analysis of SIFT, SURF, KAZE, AKAZE, ORB, and BRISK. In: 2018 International Conference on Computing, Mathematics and Engineering Technologies (iCoMET), pp 1–10. https://doi.org/10.1109/ICOMET.2018.8346440

  28. Kumar NKS, Malarvizhi N (2020) Bi-directional LSTM–CNN combined method for sentiment analysis in part of speech tagging (PoS). Int J Speech Technol 23(373–380)

  29. Kumar P, Gauba H, Roy PP, Dogra DP (2017) A multimodal framework for sensor based sign language recognition.Neurocomputing 259:21–38. https://doi.org/10.1016/j.neucom.2016.08.132

  30. Alzubaidi L et al (2021) Review of deep learning: concepts, CNN architectures, challenges, applications, future directions. J Big Data 8:53. https://doi.org/10.1186/s40537-021-00444-8

  31. Li Y, Cai Y, Malekian R, Wang H, Sotelo MA, Li Z (2021) Creating navigation map in semi-open scenarios for intelligent vehicle localization using multi-sensor fusion. Exp Syst Appl Volume: 184:1–12

    Article  Google Scholar 

  32. Li T, Li J, Liu J, Huang M, Chen YW, Bhatti UA (2022) Robust watermarking algorithm for medical images based on log-polar transform. EURASIP J Wireless Commun Netw Volume: 24:1–11

    Google Scholar 

  33. Likhar P, Bhagat NK, Rathna GN (2020) “Deep learning methods for Indian sign language recognition”,10th International Conference on Consumer Electronics (ICCE-Berlin)

  34. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60:91–110. https://doi.org/10.1023/B:VISI.0000029664.99615.94

  35. Mittal A, Kumar P, Roy PP, Balasubramanian R, Chaudhuri BB (2019) A modified-LSTM model for continuous sign language recognition using leap motion. IEEE Sens J 19(16):1–8

    Article  Google Scholar 

  36. Nanni L, Ghidoni S, Brahnam S (2017) Handcrafted vs. non-handcrafted features for computer vision classification,Pattern Recogn 71:158–172.https://doi.org/10.1016/j.patcog.2017.05.025

  37. Nawaz SA, Li J, Bhatti UA, Bazai SU, Zafar A, Bhatti MA, Mehmood A, Ain Q, Shoukat MU (2021) A hybrid approach to forecast the COVID-19 epidemic trend. Plos One 16(10):1–16

  38. Rastgoo R, Kiani K, Escalera S (2020) Hand sign language recognition using multi-view hand skeleton. Expert Syst Appl 150:113336.https://doi.org/10.1016/j.eswa.2020.113336

  39. Rastgoo R, Kiani K, Escalera S (2021) Sign language recognition: a deep survey. Expert Syst Appl 164:113794.https://doi.org/10.1016/j.eswa.2020.113794

  40. Rastgoo R, Kiani K, Escalera S (2021) Sign language recognition: a deep survey. Expert Syst Appl, [ISSN: 0957-4174] 164(113794):1–27

  41. Saraee E, Jalal M, Betke M (2020) Visual complexity analysis using deep intermediate-layer features. Comput Vis Image Underst 195:1–17

    Article  Google Scholar 

  42. Shaik KB, Ganesan P, Kalist V, Sathish BS, Jenitha JM (2015) “Comparative Study of Skin Color Detection and Segmentation in HSV and YCbCr Color Space”, Int Conf Recent Trends Computing (ICRTC), pp.41–48

  43. Singh DK (2021) “3D-CNN based Dynamic Gesture Recognition for Indian Sign Language Modeling”, Int Conf AI Comput Linguist, pp. 76–83

  44. Sonare B, Padgal A, Gaikwad Y, Patil A (2021) “Video-Based Sign Language Translation System Using Machine Learning”, 2nd International Conference for Emerging Technology (INCET), pp. 1–4

  45. Sridhar A, Ganesan RG, Kumar P, Khapra M (2020) INCLUDE: A Large Scale Dataset for Indian Sign Language Recognition. In: Proceedings of the 28th ACM International Conference on Multimedia. Association for Computing Machinery, Seattle, pp 1366–1375. https://doi.org/10.1145/3394171.3413528

  46. Tamang J, Nkapkop JDD, Ijaz MF, Prasad PK, Tsafack N, Saha A, Kengne J, Son Y (2021) Dynamical properties of ion-acoustic waves in space plasma and its application to image encryption. IEEE Access Volume:9:18762–18782

    Article  Google Scholar 

  47. Taskiran M, Killioglu M, Kahraman N (2018) A real-time system for recognition of american sign language by using deep learning. In: 2018 41st International Conference on Telecommunications and Signal Processing (TSP), pp 1–5. https://doi.org/10.1109/TSP.2018.8441304

  48. Venugopalan A, Reghunadhan R (2021) Applying deep neural networks for the automatic recognition of sign language words: A communication aid to deaf agriculturists. Expert Syst Appl Volume 185:1–9

    Article  Google Scholar 

  49. Wadhawan A, Kumar P (2020) Deep learning-based sign language recognition system for static signs. Neural Comput Appl Volume 32:7957–7968

    Article  Google Scholar 

  50. Zhang E, Botao X, Fangzhou C, Jinghong D, Guangfeng L, Yifei L (2019) Fusion of 2D CNN and 3D densenet for dynamic gesture recognition. Electron 8(1511):1–15

    Google Scholar 

Download references

Funding

The authors have no relevant financial or non-financial interests to disclose.

Author information

Authors and Affiliations

  1. NIT Silchar, 788010, Silchar, Assam, India

    Soumen Das, Saroj Kr Biswas & Biswajit Purkayastha

Authors
  1. Soumen Das
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Saroj Kr Biswas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Biswajit Purkayastha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Soumen Das.

Ethics declarations

Competing interests

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Das, S., Biswas, S.K. & Purkayastha, B. Automated Indian sign language recognition system by fusing deep and handcrafted feature. Multimed Tools Appl 82, 16905–16927 (2023). https://doi.org/10.1007/s11042-022-14084-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-14084-4

Keywords

Search

Navigation