Skip to main content
SpringerLink
Log in

Isolated Video-Based Arabic Sign Language Recognition Using Convolutional and Recursive Neural Networks

  • Research Article-Computer Engineering and Computer Science
  • Published:
Arabian Journal for Science and Engineering Aims and scope Submit manuscript

Abstract

For natural and meaningful communication between the deaf community and the hearing population, sign language is very important. Most of the Arab sign recognition studies have focused on the identification of the sign action based on the descriptor of the feature. However, the limitation of this traditional method is the need to choose which features are important in each particular sequence. To address this issue, we propose a novel approach based on a deep learning architecture to classify video sequences of Arabic sign language, especially Moroccan sign language. Two methods of classification are applied, namely 2D convolutional recurring neural network (2DCRNN) and 3D convolutional neural network (3DCNN). Concerning the first method, a 2DCRNN model is used to extract features with a recurring network pattern to detect the relationship between frames. The second method uses a 3DCNN model learning the spatiotemporal features out of small patches. After 2DCRNN and the 3DCNN models extracted feature, the video data are classified into various classes, using a fully connected network. The proposed approach is trained over a collection of 224 videos of five individuals performing 56 different signs. The results achieved through the fourfold cross-validation technique demonstrate the performance of the proposed approach in terms of recall, F1 score, and AUROC, with the level accuracy of 92% for 2DCRNN and 99% for 3DCNN.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Napier, J.; Leeson, L.: Sign language in action. In: Sign language in action, pp. 50–84. Springer (2016)

  2. Sandler, W.; Lillo-Martin, D.: Sign Language and Linguistic Universals. Cambridge University Press, Cambridge (2006)

    Book  Google Scholar 

  3. Yeasin, M.; Chaudhuri, S.: Visual understanding of dynamic hand gestures. Pattern Recognit. 33(11), 1805–1817 (2000)

    Article  Google Scholar 

  4. NIDCD Fact Sheet, H., Balance: American sign language. NIH Publication No. 11-4756 March 2019 (2019)

  5. Yang, S.; Zhu, Q.: Video-based chinese sign language recognition using convolutional neural network. In: 2017 IEEE 9th International Conference on Communication Software and Networks (ICCSN), pp. 929–934. IEEE (2017)

  6. Al-Fityani, K.; Padden, C.: Sign language geography in the arab world. Sign languages: a Cambridge survey 20,(2010)

  7. Esam, N.; Abbas, A.; Krause, P.: Towards empowering hearing impaired students’ skills in computing and technology. Int. J. Adv. Comput. Sci. Appl. 8(1) (2017)

  8. Abdel-Fattah, M.A.: Arabic sign language: a perspective. J. Deaf Stud. Deaf Educ. 10(2), 212–221 (2005)

    Article  Google Scholar 

  9. Gugenheimer, J.; Plaumann, K.; Schaub, F.; Di Campli San Vito, P.; Duck, S.; Rabus, M.; Rukzio, E.: The impact of assistive technology on communication quality between deaf and hearing individuals. In: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, pp. 669–682 (2017)

  10. Mohandes, M.; Deriche, M.; Liu, J.: Image-based and sensor-based approaches to arabic sign language recognition. IEEE Trans. Hum.-Mach. Syst. 44(4), 551–557 (2014)

  11. Tubaiz, N.; Shanableh, T.; Assaleh, K.: Glove-based continuous arabic sign language recognition in user-dependent mode. IEEE Trans. Hum.-Mach. Syst. 45(4), 526–533 (2015)

  12. Mohandes, M.; A-Buraiky, S.; Halawani, T.; Al-Baiyat, S.: Automation of the arabic sign language recognition. In: Proceedings. 2004 International Conference on Information and Communication Technologies: From Theory to Applications, 2004., pp. 479–480. IEEE (2004)

  13. Ibrahim, N.B.; Zayed, H.H.; Selim, M.M.: Advances, challenges and opportunities in continuous sign language recognition. J. Eng. Appl. Sci. 15(5), 1205–1227 (2020)

    Google Scholar 

  14. Elons, A.; Ahmed, M.; Shedid, H.; Tolba, M.: Arabic sign language recognition using leap motion sensor. In: 2014 9th International Conference on Computer Engineering & Systems (ICCES), pp. 368–373. IEEE (2014)

  15. Ahmed, A.A.; Aly, S.: Appearance-based arabic sign language recognition using hidden markov models. In: 2014 International Conference on Engineering and Technology (ICET), pp. 1–6. IEEE (2014)

  16. Abdi, H.; Williams, L.J.: Principal component analysis. Wiley Interdiscip. Rev. Comput. Stat. 2(4), 433–459 (2010)

    Article  Google Scholar 

  17. Heikkilä, M.; Pietikäinen, M.; Schmid, C.: Description of interest regions with local binary patterns. Pattern Recognit. 42(3), 425–436 (2009)

    Article  Google Scholar 

  18. Shanableh, T.; Assaleh, K.; Al-Rousan, M.: Spatio-temporal feature-extraction techniques for isolated gesture recognition in arabic sign language. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 37(3), 641–650 (2007)

  19. Hemayed, E.E.; Hassanien, A.S.: Edge-based recognizer for arabic sign language alphabet (ars2v-arabic sign to voice). In: 2010 International Computer Engineering Conference (ICENCO), pp. 121–127. IEEE (2010)

  20. Dahmani, D.; Larabi, S.: User-independent system for sign language finger spelling recognition. J. Vis. Commun. Image Represent. 25(5), 1240–1250 (2014)

    Article  Google Scholar 

  21. Hassan, M.; Assaleh, K.; Shanableh, T.: User-dependent sign language recognition using motion detection. In: 2016 International Conference on Computational Science and Computational Intelligence (CSCI), pp. 852–856. IEEE (2016)

  22. Alzohairi, R.; Alghonaim, R.; Alshehri, W.; Aloqeely, S.; Alzaidan, M.; Bchir, O.: Image based arabic sign language recognition system. Int. J. Adv. Comput. Sci. Appl. (IJACSA) 9(3) (2018)

  23. Soora, N.R.; Deshpande, P.S.: Review of feature extraction techniques for character recognition. IETE J. Res. 64(2), 280–295 (2018)

    Article  Google Scholar 

  24. Adams, S.; Beling, P.A.: A survey of feature selection methods for Gaussian mixture models and hidden Markov models. Artif. Intell. Rev. 52(3), 1739–1779 (2019)

    Article  Google Scholar 

  25. Deriche, M.; Aliyu, S.O.; Mohandes, M.: An intelligent arabic sign language recognition system using a pair of LMCs with GMM based classification. IEEE Sensors J. 19(18), 8067–8078 (2019)

    Article  Google Scholar 

  26. Hayani, S.; Benaddy, M.; El Meslouhi, O.; Kardouchi, M.: Arab sign language recognition with convolutional neural networks. In: 2019 International Conference of Computer Science and Renewable Energies (ICCSRE), pp. 1–4. IEEE (2019)

  27. Maraqa, M.; Abu-Zaiter, R.: Recognition of arabic sign language (arsl) using recurrent neural networks. In: 2008 First International Conference on the Applications of Digital Information and Web Technologies (ICADIWT), pp. 478–481. IEEE (2008)

  28. Li, D.; Rodriguez, C.; Yu, X.; Li, H.: Word-level deep sign language recognition from video: a new large-scale dataset and methods comparison. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1459–1469 (2020)

  29. Li, D.; Yu, X.; Xu, C.; Petersson, L.; Li, H.: Transferring cross-domain knowledge for video sign language recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6205–6214 (2020)

  30. Albanie, S.; Varol, G.; Momeni, L.; Afouras, T.; Chung, J.S.; Fox, N.; Zisserman, A.: Bsl-1k: scaling up co-articulated sign language recognition using mouthing cues. In: European Conference on Computer Vision, pp. 35–53. Springer (2020)

  31. Momeni, L.; Varol, G.; Albanie, S.; Afouras, T.; Zisserman, A.: Watch, read and lookup: learning to spot signs from multiple supervisors. In: Proceedings of the Asian Conference on Computer Vision (2020)

  32. Li, D.; Xu, C.; Yu, X.; Zhang, K.; Swift, B.; Suominen, H.; Li, H.: Tspnet: hierarchical feature learning via temporal semantic pyramid for sign language translation. arXiv:2010.05468 (2020)

  33. Renz, K.; Stache, N.C.; Albanie, S.; Varol, G.: Sign language segmentation with temporal convolutional networks. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2135–2139. IEEE (2021)

  34. Deng, L.; Yu, D.: Deep learning: methods and applications. Found. Trends Signal Process. 7(3–4), 197–387 (2014)

    Article  MathSciNet  Google Scholar 

  35. Ibrahim, N.B.; Selim, M.M.; Zayed, H.H.: An automatic arabic sign language recognition system (arslrs). J. King Saud Univ.-Comput. Inf. Sci. 30(4), 470–477 (2018)

  36. ElBadawy, M.; Elons, A.; Shedeed, H.A.; Tolba, M.: Arabic sign language recognition with 3d convolutional neural networks. In: 2017 Eighth International Conference on Intelligent Computing and Information Systems (ICICIS), pp. 66–71. IEEE (2017)

Download references

Acknowledgements

The authors acknowledge the support of Achourouk, Association of the Deaf of Ouarzazate, for sharing the database of Moroccan sign language used in this work.

Author information

Authors and Affiliations

  1. LabSIE Laboratory, Polydisciplinary Faculty of Ouarzazate, Ibn Zohr University, Ouarzazate, Morocco

    Abdelbasset Boukdir, Mohamed Benaddy & Ayoub Ellahyani

  2. National School of Applied Sciences - Safi, Cadi Ayyad University, Marrakesh, Morocco

    Othmane El Meslouhi

  3. Département d’Informatique, Université de Moncton, Moncton, Canada

    Mustapha Kardouchi

Authors
  1. Abdelbasset Boukdir
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohamed Benaddy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ayoub Ellahyani
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Othmane El Meslouhi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mustapha Kardouchi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Abdelbasset Boukdir.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Boukdir, A., Benaddy, M., Ellahyani, A. et al. Isolated Video-Based Arabic Sign Language Recognition Using Convolutional and Recursive Neural Networks. Arab J Sci Eng 47, 2187–2199 (2022). https://doi.org/10.1007/s13369-021-06167-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13369-021-06167-5

Keywords

Search

Navigation