Skip to main content
SpringerLink
Log in

A novel feature fusion technique for robust hand gesture recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Gesture recognition has emerged as a crucial field of research due to its wide range of applications, such as human-computer interaction, sign language interpretation, and virtual reality. In this research work, a robust framework for hand gesture recognition is developed. The publicly available dataset, i.e., the Multi-Modal Hand Gesture dataset, is used. To achieve effective classification of hand gestures, a series of preprocessing techniques are employed to enhance the image quality. The Haar wavelet transform (HWT) and Local Binary Pattern (LBP) techniques are utilized for feature fusion, extracting both global and local texture information. This fusion of features maximizes the discriminative power of the classification model. In addition, the original AlexNet model architecture is modified, now referred to as the AlexNet-11 Layer model, by incorporating additional layers that effectively learn the patterns and fused features. As a result, the proposed approach achieves a remarkable accuracy of 97.38%, outperforming the state-of-the-art methods, which achieved accuracies of 96.5%, 96% and 82%. This significant improvement showcases the effectiveness of the proposed approach in advancing the field of gesture recognition and image classification.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data Availibility Statement

Data will be made available upon reasonable request.

References

  1. Kasapbaşi A, Elbushra AEA, Omar A-H, Yilmaz A (2022) DeepASLR: a CNN based human computer interface for American sign language recognition for hearing-impaired individuals. Comput Methods and Programs in Biomedicine Update 2:100048

    Article  Google Scholar 

  2. Sarma D, Bhuyan MK (2021) Methods, databases and recent advancement of vision-based hand gesture recognition for HCI systems: a review. SN Comput Sci 2(6):436

    Article  Google Scholar 

  3. Wolfert P, Robinson N, Belpaeme T (2022) A review of evaluation practices of gesture generation in embodied conversational agents. IEEE Trans Hum-Mach Syst 52(3):379–389

    Article  Google Scholar 

  4. Bharti S, Balmik A, Nandy A (2023) Novel error correction-based key frame extraction technique for dynamic hand gesture recognition. Neural Computing and Appl 35(28):21165–21180

    Article  Google Scholar 

  5. Sen A, Mishra TK, Dash R (2022) A novel hand gesture detection and recognition system based on ensemble-based convolutional neural network. Multimedia Tools Appl 81(28):40043–40066

    Article  Google Scholar 

  6. DelPreto J, Hughes J, D’Aria M, de Fazio M, Rus D (2022) A wearable smart glove and its application of pose and gesture detection to sign language classification. IEEE Robot Autom Lett 7(4):10589–10596

    Article  Google Scholar 

  7. Ivani AS, Giubergia A, Santos L, Geminiani A, Annunziata S, Caglio A, Olivieri I, Pedrocchi A (2022) A gesture recognition algorithm in a robot therapy for ASD children. Biomedical Signal Processing and Control 74:103512

    Article  Google Scholar 

  8. Zhang W, Wang J, Lan F (2020) Dynamic hand gesture recognition based on short-term sampling neural networks. IEEE/CAA J Autom Sin 8(1):110–120

    Article  Google Scholar 

  9. Hakim NL, Shih TK, Kasthuri Arachchi SP, Aditya W, Chen Y-C, Lin C-Y (2019) Dynamic hand gesture recognition using 3DCNN and LSTM with FSM context-aware model. Sensors 19(24):5429

    Article  Google Scholar 

  10. Mazhar O, Navarro B, Ramdani S, Passama R, Cherubini A (2019) A real-time human-robot interaction framework with robust background invariant hand gesture detection. Robot Comput-Integr Manuf 60:34–48

    Article  Google Scholar 

  11. Ur Rehman M, Ahmed F, Attique Khan M, Tariq U, Abdulaziz Alfouzan F, Alzahrani NM, Ahmad J (2021) Dynamic hand gesture recognition using 3D-CNN and LSTM networks. Computers, Materials & Continua 70(3)

  12. Kopuklu O, Gunduz A, Kose N, Rigoll G (2019) Real-time hand gesture detection and classification using convolutional neural networks. In: 2019 14th IEEE International conference on automatic face & gesture recognition (FG 2019), pp 1–8, IEEE

  13. Al Farid F, Hashim N, Abdullah J, Bhuiyan MR, Shahida Mohd Isa WN, Uddin J, Haque MA, Husen MN (2022) A structured and methodological review on vision-based hand gesture recognition system. J Imaging 8(6):153

  14. Subburaj S, Murugavalli S (2022) Survey on sign language recognition in context of vision-based and deep learning. Measurement: Sensors 23:100385

    Google Scholar 

  15. Raghuveera T, Deepthi R, Mangalashri R, Akshaya R (2020) A depth-based Indian sign language recognition using microsoft kinect. Sādhanā 45:1–13

    Article  Google Scholar 

  16. Li J, Li C, Han J, Shi Y, Bian G, Zhou S (2022) Robust hand gesture recognition using HOG-9ULBP features and SVM model. Electronics 11(7):988

    Article  Google Scholar 

  17. Zhou W, Xiao Y, Yan W, Yu L (2023) CMPFFNET: cross-modal and progressive feature fusion network for RGB-D indoor scene semantic segmentation. IEEE Trans Autom Sci Eng

  18. Liu Z, Wang L, Wen Z, Li K, Pan Q (2022) Multilevel scattering center and deep feature fusion learning framework for SAR target recognition. IEEE Trans Geosci Remote Sens 60:1–14

    Google Scholar 

  19. Rajan RG, Leo MJ (2020) American sign language alphabets recognition using hand crafted and deep learning features. In: 2020 International conference on inventive computation technologies (ICICT), pp 430–434, IEEE

  20. Sharma S, Singh S (2021) Vision-based hand gesture recognition using deep learning for the interpretation of sign language. Expert Syst Appl 182:115657

    Article  Google Scholar 

  21. Li G, Tang H, Sun Y, Kong J, Jiang G, Jiang D, Tao B, Xu S, Liu H (2019) Hand gesture recognition based on convolution neural network. Clust Comput 22(2):2719–2729

    Article  Google Scholar 

  22. Gadekallu TR, Alazab M, Kaluri R, Maddikunta PKR, Bhattacharya S, Lakshmanna K (2021) Hand gesture classification using a novel CNN-crow search algorithm. Complex Intell Syst 7:1855–1868

    Article  Google Scholar 

  23. Ameur S, Khalifa AB, Bouhlel MS (2020) A novel hybrid bidirectional unidirectional LSTM network for dynamic hand gesture recognition with leap motion. Entertainment Computing 35:100373

    Article  Google Scholar 

  24. Roumiassa F, Agab SE, Chelali FZ (2022) Hand gesture recognition system based on textural features. In: 2022 2nd International conference on advanced electrical engineering (ICAEE), pp 1–6, IEEE

  25. Avola D, Cinque L, Emam E, Fontana F, Foresti GL, Marini MR, Pannone D (2023) Hand gesture recognition exploiting handcrafted features and LSTM. In: International conference on image analysis and processing, pp 500–511, Springer

  26. Ansar H, Ksibi A, Jalal A, Shorfuzzaman M, Alsufyani A, Alsuhibany SA, Park J (2022) Dynamic hand gesture recognition for smart lifecare routines via K-Ary tree hashing classifier. Appl Sci 12(13):6481

    Article  Google Scholar 

  27. Abavisani M, Joze HRV, Patel VM (2019) Improving the performance of unimodal dynamic hand-gesture recognition with multimodal training. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1165–1174

  28. Mantecón T, Del-Blanco CR, Jaureguizar F, García N (2019) A real-time gesture recognition system using near-infrared imagery. PloS ONE 14(10):e0223320

    Article  Google Scholar 

  29. Balmik A, Barik S, Jha M, Nandy A (2023) A vision-based litter detection and classification using SSD mobilenetv2. In: 2023 10th International conference on signal processing and integrated networks (SPIN), pp 180–185, IEEE

  30. Chung Y-L, Chung H-Y, Tsai W-F (2020) Hand gesture recognition via image processing techniques and deep CNN. J Intell Fuzzy Syst 39(3):4405–4418

    Article  MathSciNet  Google Scholar 

  31. Khan RZ, Ibraheem NA (2012) Hand gesture recognition: a literature review. Int J Artif Intell Appl 3(4):161

    Article  Google Scholar 

  32. Balmik A, Kumar A, Nandy A (2021) Efficient face recognition system for education sectors in COVID-19 pandemic. In: 2021 12th International conference on computing communication and networking technologies (ICCCNT), pp 1–8, IEEE

  33. Oudah M, Al-Naji A, Chahl J (2020) Hand gesture recognition based on computer vision: a review of techniques. J Imaging 6(8):73

    Article  Google Scholar 

  34. Kaur S, Kaur M (2018) Image sharpening using basic enhancement techniques. Int J Res Eng Sci Manag

  35. Balmik A, Barik S, Nandy A (2023) A robust object recognition using modified YOLOv5 neural network. In: 2023 10th International conference on signal processing and integrated networks (SPIN), pp 462–467, IEEE

  36. Stanković RS, Falkowski BJ (2003) The Haar wavelet transform: its status and achievements. Comput Electr Eng 29(1):25–44

    Article  Google Scholar 

  37. Karis MS, Razif NRA, Ali NM, Rosli MA, Aras MSM, Ghazaly MM (2016) Local binary pattern (LBP) with application to variant object detection: a survey and method. In: 2016 IEEE 12th international colloquium on signal processing & its applications (CSPA), pp 221–226, IEEE

  38. Krizhevsky A, Sutskever I, Hinton GE (2017) ImageNet classification with deep convolutional neural networks. Commun ACM 60(6):84–90

    Article  Google Scholar 

  39. Imantoko I, Hermawan A, Avianto D (2021) Comparative analysis of support vector machine and k-nearest neighbors with a pyramidal histogram of the gradient for sign language detection. Matrix: Jurnal Manajemen Teknologi dan Informatika 11(2):107–118

Download references

Author information

Author notes

  1. Sunanda and Archana Balmik These authors contributed equally to this work.

Authors and Affiliations

  1. Machine Intelligence & Bio-Motion Laboratory, Department of Computer Science & Engineering, National Institute of Technology Rourkela, Rourkela, India

    Sunanda, Archana Balmik & Anup Nandy

Authors
  1. Sunanda
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Archana Balmik
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anup Nandy
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Archana Balmik.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sunanda, Balmik, A. & Nandy, A. A novel feature fusion technique for robust hand gesture recognition. Multimed Tools Appl (2024). https://doi.org/10.1007/s11042-024-18173-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11042-024-18173-4

Keywords

Search

Navigation