Skip to main content
SpringerLink
Log in

HyFiNet: Hybrid feature attention network for hand gesture recognition

  • 1215: Multimodal Interaction and IoT Applications
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper, we propose a portable CNN network: a hybrid feature attention network (HyFiNet) for precise hand gesture recognition. HyFiNet is designed by stacking four multi-scale refined edge extraction modules (REEMs). The REEM module is introduced to capture the refined edge information of hand gestures by incorporating hybrid feature attention (HyAttention) block. The HyAttention block is intended to focus on efficient salient features from multi-receptive fields and acquire knowledge of discriminable semantic structure for hand poses. As a resultant, multi-scale feature and hybrid feature attention mechanisms cohesively improve the performance of hand gesture recognition with a minimum computational cost. The efficiency of the proposed network is validated using six benchmark datasets: MUGD, Finger Spelling, NUS-I, NUS-II, HGR-I and Triesch, by adopting two validation schemes: person dependent and person independent. Furthermore, seven supplementary experiments are also performed for an ablation study to analyze the effectiveness of each module in the proposed network. The experimental results and visual representation indicate a substantial increase in accuracy compared to the existing state-of-the-art networks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Abavisani Mahdi, Joze Hamid Reza Vaezi, Patel Vishal M (2019) Improving the performance of unimodal dynamic hand-gesture recognition with multimodal training. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p 1165–1174

  2. Abdelnasser H, Youssef M, Harras KA (2015) WiGest: A ubiquitous WiFi-based gesture recognition system. In: 2015 IEEE Conference on Computer Communications (INFOCOM), Kowloon, pp 1472–1480. https://doi.org/10.1109/INFOCOM.2015.7218525

  3. Abdessamad Elboushaki, Hannane Rachida, Afdel Karim, Koutti Lahcen (2020) MultiD-CNN: A multi-dimensional feature learning approach based on deep convolutional networks for gesture recognition in RGB-D image sequences. Expert Syst Appl 139:112829

    Article  Google Scholar 

  4. Abdulhussein Abdulwahab A, Raheem Firas A (2020) Hand Gesture Recognition of Static Letters American Sign Language (ASL) Using Deep Learning. Eng Technol J 38(6A):926–937

    Article  Google Scholar 

  5. Adithya V, Rajesh R (2020) A Deep Convolutional Neural Network Approach for Static Hand Gesture Recognition. Procedia Comput Sci 171:2353–2361

    Article  Google Scholar 

  6. Al-Hammadi Muneer, Muhammad Ghulam, Abdul Wadood, Alsulaiman Mansour, Bencherif Mohamed A, Mekhtiche Mohamed Amine (2020) Hand Gesture Recognition for Sign Language Using 3DCNN. IEEE Access 8:79491–79509

    Article  Google Scholar 

  7. Bao Peijun, Maqueda Ana I, del-Blanco Carlos, Garcia Narciso (2017) Tiny hand gesture recognition without localization via a deep convolutional network. IEEE Trans Consum Electron 63(3):251–257

    Article  Google Scholar 

  8. Barczak ALC, Reyes NH, Abastillas M, Piccio A, Susnjak T (2011) A new 2D static hand gesture colour image dataset for ASL gestures. Res Lett Inf Math Sci 15:12–20

  9. Bhaumik G, Verma M, Govil MC, Vipparthi SK (2020) EXTRA: An Extended Radial Mean Response Pattern for Hand Gesture Recognition. In: 2020 International Conference on Communication and Signal Processing (ICCSP), Chennai, India, pp. 0640–0645. https://doi.org/10.1109/ICCSP48568.2020.9182207

  10. Cheok Ming Jin, Omar Zaid, Jaward Mohamed Hisham (2019) A review of hand gesture and sign language recognition techniques. Int J Mach Learn Cybernetics 10(1):131–153

    Article  Google Scholar 

  11. Chevtchenko Sergio F, Vale Rafaella F, Macario Valmir, Cordeiro Filipe R (2018) A convolutional neural network with feature fusion for real-time hand posture recognition. Appl Soft Comput 73:748–766

    Article  Google Scholar 

  12. Ding Youdong, Pang Haibo, Wu Xuechun, Lan Jianliang (2011) Recognition of hand-gestures using improved local binary pattern. In: 2011 International Conference on Multimedia Technology, Hangzhou, pp 3171–3174. https://doi.org/10.1109/ICMT.2011.6001919

  13. Fang Bin, Lv Qin, Shan Jianhua, Sun Fuchun, Liu Huaping, Guo Di, Zhao Yize (2019) Dynamic Gesture Recognition Using Inertial Sensors-based Data Gloves. In: 2019 IEEE 4th International Conference on Advanced Robotics and Mechatronics (ICARM), IEEE, pp 390–395

  14. Flores C Jose L, Gladys Cutipa AE, Lauro Enciso R (2017) Application of convolutional neural networks for static hand gestures recognition under different invariant features. In: 2017 IEEE XXIV International Conference on Electronics, Electrical Engineering and Computing (INTERCON), IEEE, p 1–4

  15. Gamal Heba M, HM Abdul-Kader, Elsayed A. Sallam (2013) Hand gesture recognition using fourier descriptors. In: 2013 8th International Conference on Computer Engineering and Systems (ICCES), IEEE, pp 274–279

  16. Ganguly Biswarup, Vishwakarma Priyanka, Biswas Shreya (2020) Kinect Sensor Based Single Person Hand Gesture Recognition for Man-Machine Interaction. In: Computational Advancement in Communication Circuits and Systems, Springer, Singapore, pp 139–144

  17. Gupta Hari Prabhat, Chudgar Haresh S, Mukherjee Siddhartha, Dutta Tanima, Sharma Kulwant (2016) A continuous hand gestures recognition technique for human-machine interaction using accelerometer and gyroscope sensors. IEEE Sens J 16(16):6425–6432

    Article  Google Scholar 

  18. Hang Su, Ovur Salih Ertug, Zhou Xuanyi, Qi Wen, Ferrigno Giancarlo, De Momi Elena (2020) Depth vision guided hand gesture recognition using electromyographic signals. Adv Robot 1–13

  19. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, p 770–778

  20. Kawulok M, Kawulok J, Nalepa J, and Smolka B. (2014) Self-adaptive algorithm for segmenting skin regions. EURASIP Journal on Advances in Signal Processing, vol. 2014, no. 170.

  21. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint https://arxiv.org/abs/1704.04861v1

  22. Huang Deng-Yuan, Wu-Chih Hu, Chang Sung-Hsiang (2011) Gabor filter-based hand-pose angle estimation for hand gesture recognition under varying illumination. Expert Syst Appl 38(5):6031–6042

    Article  Google Scholar 

  23. Huang Gao, Liu Zhuang, Van Der Maaten Laurens, Weinberger Kilian Q (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, p 4700–4708

  24. Islam Md Zahirul, Hossain Mohammad Shahadat, ul Islam Raihan, Andersson Karl (2019) Static hand gesture recognition using convolutional neural network with data augmentation. In: 2019 Joint 8th International Conference on Informatics, Electronics and Vision (ICIEV) and 2019 3rd International Conference on Imaging, Vision and Pattern Recognition (icIVPR), IEEE, pp 324–329

  25. Joze Hamid Reza Vaezi, Shaban Amirreza, Iuzzolino Michael L, Koishida Kazuhito (2020) MMTM: Multimodal Transfer Module for CNN Fusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 13289–13299

  26. Khan Asifullah, Sohail Anabia, Zahoora Umme, Qureshi Aqsa Saeed (2020) A survey of the recent architectures of deep convolutional neural networks. Artif Intell Rev 1–62

  27. Köpüklü Okan, Gunduz Ahmet, Kose Neslihan, Rigoll Gerhard (2019) Real-time hand gesture detection and classification using convolutional neural networks. In: 14th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2019), IEEE, pp 1–8

  28. Krisandria Kevin Nathanael, Dewantara Bima Sena Bayu, Pramadihanto Dadet (2019) HOG-based Hand Gesture Recognition Using Kinect. In: 2019 International Electronics Symposium (IES), IEEE, pp 254–259

  29. Lahiani Houssem, Neji Mahmoud (2018) Hand gesture recognition method based on HOG-LBP features for mobile devices. Procedia Comput Sci 126:254–263

    Article  Google Scholar 

  30. Li Yu-Ting, Wachs Juan P (2014) HEGM: A hierarchical elastic graph matching for hand gesture recognition. Pattern Recognit 47(1):80–88

    Article  Google Scholar 

  31. Li Yuan, Xinggang Wang, Wenyu Liu, Bin Feng (2018) Deep attention network for joint hand gesture localization and recognition using static RGB-D images. Inf Sci 441:66–78

    Article  Google Scholar 

  32. Liu Kui, Kehtarnavaz Nasser (2016) Real-time robust vision-based hand gesture recognition using stereo images. J Real Time Image Process 11(1):201–209

    Article  Google Scholar 

  33. Liu Fenglin, Zeng Wei, Yuan Chengzhi, Wang Qinghui, Wang Ying (2019) Kinect-based hand gesture recognition using trajectory information, hand motion dynamics and neural networks. Artif Intell Rev 52(1):563–583

    Article  Google Scholar 

  34. Lu Danling, Yu Yuanlong, Liu Huaping (2016) Gesture recognition using data glove: An extreme learning machine method. In: 2016 IEEE International Conference on Robotics and Biomimetics (ROBIO), IEEE, pp 1349–1354

  35. Misra Arindam, Abe Takashi, Deguchi Koichiro (2011) Hand gesture recognition using histogram of oriented gradients and partial least squares regression. In: MVA, pp 479–482

  36. Mohanty A, Rambhatla SS, Sahay RR (2017) Deep gesture: static hand gesture recognition using CNN. In: Proceedings of International Conference on Computer Vision and Image Processing, Springer, pp 449–461

  37. Muthukumar K, Poorani S, Gobhinath S (2017) Vision based hand gesture recognition for Indian sign languages using local binary patterns with support vector machine classifier. Adv Nat Appl Sci 11(6 SI):314–322

    Google Scholar 

  38. Naidu Chetna, Ghotkar Archana (2016) Hand gesture recognition using leap motion controller. Int J Sci Res 2319–7064

  39. Oyedotun Oyebade K, Khashman Adnan (2017) Deep learning in vision-based static hand gesture recognition. Neural Comput Appl 28(12):3941–3951

    Article  Google Scholar 

  40. Pan Tse-Yu, Tsai Wan-Lun, Chang Chen-Yuan, Yeh Chung-Wei, Hu Min-Chun (2020) A Hierarchical Hand Gesture Recognition Framework for Sports Referee Training-Based EMG and Accelerometer Sensors. IEEE Trans Cybernetics

  41. Pramod Kumar P, Prahlad Vadakkepat, Loh Ai Poh (2017) The NUS hand posture datasets I. ScholarBank@NUS Repository.[Dataset] https://doi.org/10.25540/6PR9-R5HS

  42. Pisharady Pramod Kumar, Vadakkepat Prahlad, Loh Ai Poh (2013) Attention Based Detection and Recognition of Hand Postures Against Complex Backgrounds. Int J Comput Vis 101(3):403–419

    Article  Google Scholar 

  43. ASL Finger Spelling Dataset (2017) http://personal.ee.surrey.ac.uk/Personal/N.Pugeault/index.php?section=FingerSpellingDataset. Accessed on 10 Jan. 2019

  44. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) Mobilenetv 2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4510–4520

  45. Sedmidubsky Jan, Elias Petr, Budikova Petra, Zezula Pavel (2021) Content-Based Management of Human Motion Data: Survey and Challenges. IEEE Access 9:64241–64255

    Article  Google Scholar 

  46. Simão Miguel, Neto Pedro, Gibaru Olivier (2016) Natural control of an industrial robot using hand gesture recognition with neural networks. In: IECON 2016-42nd Annual Conference of the IEEE Industrial Electronics Society, IEEE, pp. 5322–5327

  47. Song W, Han Q, Lin Z, Yan N, Luo D, Liao Y, Zhang M, Wang Z, Xie X, Wang A, Chen Y (2019) Design of a flexible wearable smart sEMG recorder integrated gradient boosting decision tree based hand gesture recognition. IEEE Trans Biomed Circuits Syst 13(6):1563–1574

    Article  Google Scholar 

  48. Sykora Peter, Kamencay Patrik, Hudec Robert (2014) Comparison of SIFT and SURF methods for use on hand gesture recognition based on depth map. Aasri Procedia 9:19–24

    Article  Google Scholar 

  49. Szegedy C, Ioffe S, Vanhoucke V, Alemi A (2017) Inception-v4, inception-resnet and the impact of residual connections on learning. In: Thirty-first AAAI conference on artificial intelligence

  50. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2818–2826

  51. Tang Hao, Liu Hong, Xiao Wei, Sebe Nicu (2019) Fast and robust dynamic hand gesture recognition via key frames extraction and feature fusion. Neurocomputing 331:424–433

    Article  Google Scholar 

  52. Triesch Jochen, von der Malsburg Christoph (1996) Robust Classification of Hand Postures against Complex Backgrounds. In: Proceedings of the Second International Conference on Automatic Face and Gesture Recognition, IEEE Computer Society Press, Killington, Vermont, USA, p 170–175, 14–16 October 1996

  53. Wei Wentao, Yongkang Wong YuDu, Yu Hu, Kankanhalli Mohan, Geng Weidong (2019) A multi-stream convolutional neural network for sEMG-based gesture recognition in muscle-computer interface. Pattern Recognit Lett 119:131–138

    Article  Google Scholar 

  54. Yamashita Takayoshi, Watasue Taro (2014) Hand posture recognition based on bottom-up structured deep convolutional neural network with curriculum learning. In: 2014 IEEE International Conference on Image Processing (ICIP), IEEE, p 853–857

  55. YingYing, C. H. E. N., Jie Yang, Yan Wang, Jian Liu, and Marco Gruteser.(2021) Device-free activity identification using fine grained WiFi signatures. U.S. Patent 10,992,774, issued April 27.

  56. Yu Bin, Luo Zhiming, Wu Huangbin, Li Shaozi (2020) Hand gesture recognition based on attentive feature fusion. Concurr Comput 32(22):e5910

  57. Zeng Wei, Wang Cong, Wang Qinghui (2018) Hand gesture recognition using Leap Motion via deterministic learning. Multimedia Tools and Applications 77(21):28185–28206

    Article  Google Scholar 

  58. Zhan F (2019) Hand gesture recognition with convolution neural networks. In: 2019 IEEE 20th International Conference on Information Reuse and Integration for Data Science (IRI), IEEE, pp 295–298

  59. Zoph Barret, Vasudevan Vijay, Shlens Jonathon, and Le Quoc V (2018) Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, p 8697–8710

Download references

Author information

Authors and Affiliations

  1. CSE Dept, National Institute of Technology Sikkim, Sikkim, India

    Gopa Bhaumik & Mahesh Chandra Govil

  2. CSE Dept, Malaviya National Institute of Technology Jaipur, Jaipur, India

    Monu Verma & Santosh Kumar Vipparthi

Authors
  1. Gopa Bhaumik
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Monu Verma
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mahesh Chandra Govil
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Santosh Kumar Vipparthi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gopa Bhaumik.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bhaumik, G., Verma, M., Govil, M.C. et al. HyFiNet: Hybrid feature attention network for hand gesture recognition. Multimed Tools Appl 82, 4863–4882 (2023). https://doi.org/10.1007/s11042-021-11623-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11623-3

Keywords

Search

Navigation