Skip to main content
SpringerLink
Log in

Real-time computer vision-based gestures recognition system for bangla sign language using multiple linguistic features analysis

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Dynamic gestures recognition is a challenging task in the computer vision research area and still difficult to incorporate all linguistic features of a sign language in case of recognition. The proposed method has addressed only three linguistic features (hand-shape, position, and movement) among five more features for a real-time computer vision-based gestures recognition system for Bangla sign language (GRS-BdSL). The system uses Normalized Outer Boundary Vector (NOBV) and proposed Binary Window-Grid Vector (BWGV) of binary hand gestures to classify hand-shapes. Hand position is identified by using the proposed model of hand Position Mapping Filter (PMF). In parallel, the system tracks the movement path of hand-shape using the Adaptive Kalman Filter (AKF). After getting those three linguistic features, the system converts these into corresponding encoding patterns which are used to train and test the system. The proposed system recognizes each gesture by measuring the maximum Inter-Correlation Coefficient (ICC) between the encoding patterns of the test and pre-trained gestures. The system is trained and tested for 100 gestures of Bangla sign language (BdSL) achieving a mean recognition accuracy of 95.43% with the computational costs of 56.013 ms/f. We have also compared the performance of the proposed method with existing methods and have demonstrated that the proposed method has outperformed them under a similar experimental setup.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

Data Availability

Raw data were generated at Center for Disability Development (CDD) [12, 13]. Derived data supporting the findings of this study are available from the responsible person of CDD on request.

Code Availability

Raw codes of the implemented system will be available from the corresponding author (Muhammad Aminur Rahaman) on request.

References

  1. Ahmed ST, Akhand MAH (2016) Bangladeshi sign language recognition using fingertip position. In: 2016 International Conference on Medical Engineering, Health Informatics and Technology (MediTec), pp 1–5

  2. Al-Hammadi M, Muhammad G, Abdul W, Alsulaiman M, Bencherif MA, Alrayes TS, Mekhtiche MA (2020) Deep learning-based approach for sign language gesture recognition with efficient hand gesture representation. IEEE Access 8:192527–192542

    Article  Google Scholar 

  3. Al-Hammadi M, Muhammad G, Abdul W, Alsulaiman M, Bencherif MA, Mekhtiche MA (2020) Hand gesture recognition for sign language using 3dcnn. IEEE Access 8:79491–79509

    Article  Google Scholar 

  4. Anami BS, Bhandage VA (2018) Combined hu moments, orientation knowledge, and grid intersections feature based identification of bharatanatyam mudra images. Pattern Analysis and Applications

  5. Asaari MSM, Rosdi BA, Suandi SA (2015) Adaptive kalman filter incorporated eigenhand (akfie) for real-time hand tracking system. Multimed Tools Appl 74(21):9231–9257

    Article  Google Scholar 

  6. Aziz KE, Wadud A, Sultana S, Hussain MA, Bhuiyan A (2017) Bengali sign language recognition using dynamic skin calibration and geometric hashing. In: 2017 6th International Conference on Informatics, Electronics and Vision & 2017 7th International Symposium in Computational Medical and Health Technology (ICIEV-ISCMHT), vol 00, pp 1–5

  7. Battison R (1978) Lexical borrowing in american sign language. Silver spring MD: linstok press

  8. Begum S, Hasanuzzaman M (2009) Computer vision-based bangladeshi sign language recognition system. In: Proc of the 12th International Conference on Computer and Information Technology (ICCIT), pp 414–419, Dhaka, Bangladesh

  9. Bird JJ, Ekárt A, Faria DR (2020) British sign language recognition via late fusion of computer vision and leap motion with transfer learning to american sign language. Sensors, 20(18)

  10. Boukir S, Chenevière F (2004) Compression and recognition of dance gestures using a deformable model. Pattern Anal Appl 7(3):308–316

    Article  MathSciNet  Google Scholar 

  11. Boulares M, Jemni M (2017) Automatic hand motion analysis for the sign language space management. Pattern Analysis and Applications

  12. CD-1 Elementary (2002) video clips and graphics with descriptions. Centre for Disability in Development, A-18/6, Genda, Savar, Dhaka 1340, Bangladesh

  13. CD-2 Advanced (2002) video clips and graphics with descriptions. Centre for Disability in Development, A-18/6, Genda, Savar, Dhaka 1340, Bangladesh

  14. CDD (2002) Manual on Sign Supported Bangla. Center for disability in development (CDD), Dhaka, Bangladesh

  15. Cuxac C (1997) Iconicité et mouvement des signes en langue des signes française, le mouvement. Actes de la sixième Ecole d’eté de l’Association pour la Recherche Cognitive (ARC), Paris (-), pp 205–218

  16. Faraway JJ (2001) Modeling hand trajectories during reaching motions. Technical Report 383, Department of Statistics University of Michigan

  17. Fendri E, Boukhriss RR, Hammami M (2017) Fusion of thermal infrared and visible spectra for robust moving object detection. Pattern Anal Appl 20 (4):907–926

    Article  MathSciNet  Google Scholar 

  18. Guo W, Hu W, Liu C, Lu T (2019) 3d object recognition from cluttered and occluded scenes with a compact local feature. Mach Vis Appl 30:763–783, 4

    Article  Google Scholar 

  19. Hossain S, Sarma D, Mittra T, Alam MN, Saha Is, Johora FT (2020) Bengali hand sign gestures recognition using convolutional neural network. In: 2020 2nd International Conference on Inventive Research in Computing Applications (ICIRCA), pp 636–641

  20. Hua G, Li L, Liu S (2020) Multipath affinage stacked—hourglass networks for human pose estimation. Front Comput Sci 14(4):144701–

  21. IVT (1998) La langue des Signes. International Visual Theatre (IVT), 1 2 et 3 edition

  22. Islalm MDS, Rahman MDM, Rahman MDH, Arifuzzaman MD, Sassi R, Aktaruzzaman MD (2019) [ieee 2019 international conference on innovation and intelligence for informatics, computing, and technologies (3ict) - sakhier, bahrain (2019.9.22-2019.9.23)] 2019 international conference on innovation and intelligence for informatics, computing, and technologies (3ict) - recognition bangla sign language using convolutional neural network

  23. Jasim M, Zhang T, Hasanuzzaman M (2014) A real-time computer vision-based static and dynamic hand gesture recognition system. Int J Image Graphics 14(01n02):1–19

    Article  Google Scholar 

  24. Kolivand H, Joudaki S, Sunar MS, Tully D (2021) A new framework for sign language alphabet hand posture recognition using geometrical features through artificial neural network (part 1). Neural Computing and Applications, 33(10)

  25. Koller O, Forster VJ, Ney VH (2015) Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signers. Comput Vis Image Underst 141(-):108–125

    Article  Google Scholar 

  26. Le Dong, Feng Ning, Mao Mengdie, He Ling, Wang Jingjing (2017) E-grabcut: an economic method of iterative video object extraction. Front Comput Sci 11(4):649–660

  27. Li Y, Miao Q, Qi X, Ma Z, Ouyang W (2019) A spatiotemporal attention-based resc3d model for large-scale gesture recognition. Mach Vis Appl 30:875–888, 12

    Article  Google Scholar 

  28. Li Yg, Qi Z, Liu Z, Liu H, Ling M, Shi L, Liu X (2019) An embedded implementation of cnn-based hand detection and orientation estimation algorithm. Mach Vis Appl 30:1071–1082, 6

    Article  Google Scholar 

  29. Li G, Tang H, Sun Y, Kong J, Jiang G, Jiang D, Tao B, Xu S, Liu H (2019) Hand gesture recognition based on convolution neural network. Cluster Comput 22:2719–2729, 12

    Article  Google Scholar 

  30. Lu Z, Qin S, Li X, Li L, Zhang D (2019) One-shot learning hand gesture recognition based on modified 3d convolutional neural networks. Mach Vis Appl 30:8

    Article  Google Scholar 

  31. Luo Y, Celenk M (2008) A new adaptive kalman filtering method for block-based motion estimation. In: Proc of the 15th International conference on systems, signals and image processing, pp 89–92

  32. Mangla FU, Bashir A, Lali I, Bukhari AC, Shahzad B (2020) A novel key-frame selection-based sign language recognition framework for the video data. Imaging Sci J -(-):1–14

    Google Scholar 

  33. Michelle J (2018) Sign Language (ASL) Explanations. https://www.startasl.com/basic-sign-language, 2010 Online; Accessed May 25

  34. Moody B (1986) La langue des signes - Tome 2: dictionnaire bilingue élémentaire. International Visual Theatre (IVT), Editions Ellipses Paris

  35. Mukushev M, Sabyrov A, Imashev A, Koishibay K, Kimmelman V, Sandygulova A (2020) Evaluation of manual and non-manual components for sign language recognition. European Language Resources Association (ELRA) 0(0):6075–6080

    Google Scholar 

  36. Nihal RA, Rahman S, Broti NM, Deowan SA (2021) Bangla sign alphabet recognition with zero-shot and transfer learning. Pattern Recognition Letters

  37. Rahaman MA, Jasim M, Ali MH, Hasanuzzaman M (2014) Real-time computer vision-based bengali sign language recognition. In: Proc of the 17th Int Conf on Computer and Information Technology (ICCIT), pp 192–197, Dhaka, Bangladesh

  38. Rahaman MA, Jasim M, Ali MH, Hasanuzzaman M (2015) Computer vision based bengali sign words recognition using contour analysis. In: Proc of the 18th Int Conf on Computer and Information Technology (ICCIT), pp 335–340, Dhaka, Bangladesh

  39. Rahaman MA, Jasim M, Ali MH, Hasanuzzaman MD (2020) Bangla language modeling algorithm for automatic recognition of hand-sign-spelled bangla sign language. Fron Comput Sci 14(3):143302–

  40. Rahaman MA, Jasim M, Ali MH, Zhang T, Hasanuzzaman M (2018) A real-time hand-signs segmentation and classification system using fuzzy rule based rgb model and grid-pattern analysis. Fron Comput Sci 12(6):1258–1260

    Article  Google Scholar 

  41. Rahaman MA, Jasim M, Zhang T, Ali MH, Hasanuzzaman M (2015) Real-time bengali and chinese numeral signs recognition using contour matching. In: Proc of the IEEE International Conference on Robotics and Biomimetics (ROBIO), pp 1215–1220, Zhuhai, China

  42. Rinalduzzi M, De Angelis A, Santoni F, Buchicchio E, Moschitta A, Carbone P, Bellitti P, Serpelloni M (2021) Gesture recognition of sign language alphabet using a magnetic positioning system. Appl Sci 11:1–20

    Article  Google Scholar 

  43. Santa U, Tazreen Fa, Chowdhury SA (2017) Bangladeshi hand sign language recognition from video. In: IEEE 2017 20th International Conference of Computer and Information Technology (ICCIT) - Dhaka, Bangladesh, vol 00, pp 1–4

  44. Sidig AI, Mahmoud SA (2018) Trajectory based arabic sign language recognition. Int J Adv Comput Sci Appl 9(4):283–291

    Google Scholar 

  45. Starner T, Pentland A (1995) Real-time american sign language reognition from video using hidden markov models. Technical Report TR-375 M.I.T Media Laboratory Pereptual Computing Section

  46. Stoke WC, Casterline DC, Cronenberg CG (1976) Dictionary of american sign language on linguistic of american sign language on linguistic principals silver spring: Linstok press new edition edition

  47. Tabassum T, Mahmud I, Uddin MDP, Emran A, Afjal MI, Nitu AM (2020) Enhancement of single-handed bengali sign language recognition based on hog features. J Theor Appl Inf Technol 98(05):743–756

    Google Scholar 

  48. Talukder D, Jahara F (2020) Real-time bangla sign language detection with sentence and speech generation. In: 2020 23rd International Conference on Computer and Information Technology (ICCIT), pp 1–6

  49. Talukder D, Jahara F (2021) Real-time bangla sign language detection with sentence and speech generation. In: 2020 23rd International Conference on Computer and Information Technology (ICCIT), p 04

  50. World Health Organization (2021) Deafness and hearing loss. https://www.who.int/es/news-room/fact-sheets/detail/deafness-and-hearing-loss, 2019 Online; Accessed July 4

  51. Yasir F, Prasad PWC, Alsadoon A, Elchouemi A (2015) Sift based approach on bangla sign language recognition. In: Proc of the IEEE 8th International Workshop on Computational Intelligence and Applications (IWCIA), pp 35–39, Hiroshima

  52. Yasir F, Prasad PWC, Alsadoon A, Elchouemi A, Sreedharan S (2017) Bangla sign language recognition using convolutional neural network. In: 2017 International Conference on Intelligent Computing, Instrumentation and Control Technologies (ICICICT), pp 49–53

  53. Zadghorban M, Nahvi M (2018) An algorithm on sign words extraction and recognition of continuous persian sign language based on motion and shape features of hands. Pattern Anal Applic 21(2):323–335

    Article  MathSciNet  Google Scholar 

  54. Zbakh M, Haddad Z, Krahe JL (2015) An online reversed french sign language dictionary based on a learning approach for signs classification. Pattern Recogn Lett 67:28–38

    Article  ADS  Google Scholar 

  55. sourceforge.net (2015) EmguCV. http://sourceforge.net/projects/emgucv/files/latest/download Online; Accessed December 8

Download references

Funding

This work was supported in part by the Center for Research, Innovation, and Transformation (CRIT) of Green University of Bangladesh (GUB) under grant No. GUBRG/7/2021.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Green University of Bangladesh, 220/D, Begum Rokeya Sarani, Dhaka-1207, Bangladesh

    Muhammad Aminur Rahaman

  2. Department of Computer Science and Engineering, University of Dhaka, Dhaka-1000, Bangladesh

    Md. Haider Ali & Md. Hasanuzzaman

Authors
  1. Muhammad Aminur Rahaman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Md. Haider Ali
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Md. Hasanuzzaman
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors listed have contributed sufficiently to the research to be included as authors, and all those who are qualified to be authors are listed as Muhammad Aminur Rahaman, Green University of Bangladesh, Md. Haider Ali and Md. Hasanuzzaman, Department of Computer Science and Engineering, University of Dhaka, Dhaka-1000, Bangladesh.

Corresponding author

Correspondence to Muhammad Aminur Rahaman.

Ethics declarations

Ethical Statement

We confirm that all the participants who are involved in this research work especially for data collection purposes have given their ethical statements through a participant consent form.

Conflict of Interests

We wish to confirm that there are no known conflicts of interest associated with this publication and there has been no other significant financial support for this work that could have influenced its outcome.

We confirm that the manuscript has been read and approved by all named authors and that there are no other persons who satisfied the criteria for authorship but are not listed. We further confirm that the order of authors listed in the manuscript has been approved by all of us.

We confirm that we have given due consideration to the protection of intellectual property associated with this work and that there are no impediments to publication, including the timing of publication, concerning intellectual property. In so doing we confirm that we have followed the regulations of our institutions concerning intellectual property. We understand that the Corresponding Author is the sole contact for the Editorial process (including Editorial Manager and direct communications with the office). He is responsible for communicating with the other authors about progress, submissions of revisions, and final approval of proofs. We confirm that we have provided a current, correct email address which is accessible by the Corresponding Author.

To the best of our knowledge, no conflict of interest, financial or other, exists.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rahaman, M.A., Ali, M.H. & Hasanuzzaman, M. Real-time computer vision-based gestures recognition system for bangla sign language using multiple linguistic features analysis. Multimed Tools Appl 83, 22261–22294 (2024). https://doi.org/10.1007/s11042-023-15583-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15583-8

Keywords

Search

Navigation