Skip to main content
SpringerLink
Log in

RGB-D feature extraction method for hand gesture recognition based on a new fast and accurate multi-channel cartesian Jacobi moment invariants

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Due to the diversity of hand gestures uses in human computer interaction and the complexity involved by gestures, many features have been proposed, however each feature has its own drawbacks. Therefore, in this work, we propose a new set of Red, Green, Blue and Depth (RGB-D) feature extraction method based on Image Moment Invariants, named Fast and Accurate Multi-channel Cartesian Jacobi Moment Invariants for Depth (FA-MCJMID), RGB (FA-MCJMIRGB) and RGB-D (FA-MCJMIRGBD) images. We first introduce the fundamental concepts and properties to present the Multi-channel Cartesian Jacobi Moments (MCJMs). Then, we express the MCJMIs using geometric moment invariants under Rotation, Scaling and Translation (RST) transforms. Moreover, we provide the theoretical approach to enhance their numerical accuracy and improve their computational speed. Then we explore the application of our new moment invariants in hand gesture representation and recognition. Accordingly, several experiments are conducted to validate this new set of FA-MCJMI in comparison with some deep learning approches and other existing methods, on several popular hand gesture datasets, with regard to image reconstruction, invariability, numerical stability, computational complexity and recognition. The experiments demonstrate the superiority of the new FA-MCJMI set over the methods commonly used in the literature under geometric distortions, illumination variations and image occlusions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

References

  1. Aggarwal A, Sharma S, Singh K, Singh H, Kumar S (2019) A new approach for effective retrieval and indexing of medical images. Biomed Signal Process Control 50:10–34

    Article  Google Scholar 

  2. Amakdouf H, Zouhri A, El Mallahi M, Tahiri A, Chenouni D, Qjidaa H (2021) Artificial intelligent classification of biomedical color image using quaternion discrete radial Tchebichef moments. Multimed Tool Appl 80(2):3173–3192

    Article  Google Scholar 

  3. Anagha P, Baskar A (2021) An automatic histogram detection and information extraction from document images. Int J Speech Tech 24(1):77–85

    Article  Google Scholar 

  4. Benouini R, Batioua I, Elouariachi I, Zenkouar K, Zarghili A (2019) Explicit separable two dimensional moment invariants for object recognition. Procedia Comput Sci 148:409–417

    Article  Google Scholar 

  5. Benouini R, Batioua I, Zenkouar K, Najah S (2021) Fractional-order generalized Laguerre moments and moment invariants for grey-scale image analysis. IET Image Process 15(2):523–541

    Article  Google Scholar 

  6. Benouini R, Batioua I, Zenkouar K, Zahi A, Najah S, Qjidaa H (2019) Fractional-order orthogonal Chebyshev Moments and Moment Invariants for image representation and pattern recognition. Pattern Recogn 86:332–343

    Article  Google Scholar 

  7. Cai J, Luo J, Wang S, Yang S (2018) Feature selection in machine learning: a new perspective. Neurocomputing 300:70–79

    Article  Google Scholar 

  8. Camacho-Bello C, Toxqui-Quitl C, Padilla-Vivanco A, Báez-Rojas J (2014) High-precision and fast computation of Jacobi–Fourier moments for image description. JOSA A 31(1):124–134

    Article  Google Scholar 

  9. Chen B, Qi X, Sun X, Shi Y-Q (2017) Quaternion pseudo-Zernike moments combining both of RGB information and depth information for color image splicing detection. J Vis Commun Image Represent 49:283–290

    Article  Google Scholar 

  10. Cheng H, Chung SM (2016) Orthogonal moment-based descriptors for pose shape query on 3D point cloud patches. Pattern Recogn 52:397–409

    Article  Google Scholar 

  11. Chiang A, Liao SX (2015) Image analysis with legendre moment descriptors. J Comput Sci 11(1):127–136

    Article  Google Scholar 

  12. Dai X, Liu T, Shu H, Luo L (2012) Pseudo-zernike moment invariants to blur degradation and their use in image recognition. In: International conference on intelligent science and intelligent data engineering, Springer, pp 90–97

  13. Devulapalli S, Krishnan R (2021) Remote sensing image retrieval by integrating automated deep feature extraction and handcrafted features using curvelet transform. J Appl Remote Sens 15(1):016504

    Article  Google Scholar 

  14. Dickey JM (1983) Multiple functions: hypergeometric Probabilistic interpretations and statistical uses. J Am Stat Assoc 78(383):628–637

    Article  MATH  Google Scholar 

  15. El Mallahi M, El Mekkaoui J, Zouhri A, Amakdouf H, Qjidaa H (2018) Rotation scaling and translation invariants of 3D radial shifted Legendre moments. Int J Autom Comput 15(2):169–180

    Article  Google Scholar 

  16. El Ouariachi I, Benouini R, Zenkouar K, Zarghili A, El Fadili H (2022) Sign language recognition with quaternion moment invariants: a comparative study. In: Networking, intelligent systems and security, Springer, pp 737–748

  17. Elouariachi I, Benouini R, Zenkouar K, Zarghili A (2020) Robust hand gesture recognition system based on a new set of quaternion Tchebichef moment invariants. Pattern Anal Applic, 1–17

  18. Elouariachi I, Benouini R, Zenkouar K, Zarghili A, El Fadili H (2021) Explicit quaternion krawtchouk moment invariants for finger-spelling sign language recognition. In: 2020 28Th european signal processing conference (EUSIPCO), IEEE, pp 620–624

  19. Flusser J, Suk T, Zitová B (2016) 2D and 3D image analysis by moments. Wiley, Hoboken

    Book  MATH  Google Scholar 

  20. Guo L-Q, Zhu M (2011) Quaternion Fourier–Mellin moments for color images. Pattern Recogn 44(2):187–195

    Article  MATH  Google Scholar 

  21. Hao Y, Li Q, Mo H, Zhang H, Li H (2018) AMI-Net: convolution neural networks with affine moment invariants. IEEE Signal Process Lett 25(7):1064–1068

    Article  Google Scholar 

  22. Hosny KM (2007) Exact and fast computation of geometric moments for gray level images. Appl Math Comput 189(2):1214–1222

    MathSciNet  MATH  Google Scholar 

  23. Hosny KM (2014) New set of Gegenbauer moment invariants for pattern recognition applications. Arab J Sci Eng 39(10):7097–7107

    Article  MATH  Google Scholar 

  24. Hosny KM, Darwish MM (2019) New set of multi-channel orthogonal moments for color image representation and recognition. Pattern Recogn 88:153–173

    Article  Google Scholar 

  25. Hosny KM, Darwish MM, Aboelenen T (2020) New fractional-order Legendre-Fourier moments for pattern recognition applications. Pattern Recogn 103:107324

    Article  Google Scholar 

  26. Huang D-Y, Hu W-C, Chang S-H (2011) Gabor filter-based hand-pose angle estimation for hand gesture recognition under varying illumination. Expert Syst Appl 38(5):6031–6042

    Article  Google Scholar 

  27. Karakasis EG, Papakostas GA, Koulouriotis DE, Tourassis VD (2013) Generalized dual Hahn moment invariants. Pattern Recogn 46(7):1998–2014

    Article  MATH  Google Scholar 

  28. Koekoek R, Swarttouw RF The Askey-scheme of hypergeometric orthogonal polynomials and its q-analogue, arXiv: arXiv:math/9602214

  29. Lakshmi NSR, Manoharan C (2011) An automated system for classification of micro calcification in mammogram based on Jacobi moments. Int J Comput Theory Eng 3(3):431–434

    Article  Google Scholar 

  30. Li D, Kong F, Liu J, Wang Q Superpixel-Based Multiple Statistical Feature Extraction Method for Classification of Hyperspectral Images, IEEE Trans Geosci Remote Sens

  31. Lin J, Ding Y (2013) A temporal hand gesture recognition system based on hog and motion trajectory. Optik 124(24):6795–6798

    Article  Google Scholar 

  32. Monge-Alvarez J, Hoyos-Barceló C, Dahal K, Casaseca-de-la Higuera P (2018) Audio-cough event detection based on moment theory. Appl Acoust 135:124–135

    Article  Google Scholar 

  33. Mukundan R, Ong S, Lee PA (2001) Image analysis by Tchebichef moments. IEEE Trans Image Process 10(9):1357–1364

    Article  MathSciNet  MATH  Google Scholar 

  34. Mukundan R, Ramakrishnan K (1995) Fast computation of Legendre and Zernike moments. Pattern Recogn 28(9):1433–1442

    Article  MathSciNet  Google Scholar 

  35. Nwali M, Liao S (2019) A new fast algorithm to compute continuous moments defined in a rectangular region. Pattern Recogn 89:151–160

    Article  Google Scholar 

  36. Oyedotun OK, Khashman A (2017) Deep learning in vision-based static hand gesture recognition. Neural Comput Applic 28(12):3941–3951

    Article  Google Scholar 

  37. Ozcan T, Basturk A (2020) Static facial expression recognition using convolutional neural networks based on transfer learning and hyperparameter optimization. Multimed Tool Appl 79(35):26587–26604

    Article  Google Scholar 

  38. Papakostas GA, Boutalis YS, Papaodysseus C, Fragoulis DK (2006) Numerical error analysis in Zernike moments computation. Image Vis Comput 24 (9):960–969

    Article  Google Scholar 

  39. Patil SB, Sinha G (2017) Distinctive feature extraction for Indian Sign Language (ISL) gesture using scale invariant feature Transform (SIFT). J Instit Eng (India): Ser B 98(1):19–26

    Google Scholar 

  40. Pugeault N, Bowden R (2011) Spelling it out: Real-time ASL fingerspelling recognition. In: 2011 IEEE International conference on computer vision workshops (ICCV workshops), IEEE, pp 1114–1119

  41. Ren Z, Yuan J, Meng J, Zhang Z (2013) Robust part-based hand gesture recognition using kinect sensor. IEEE Trans Multimed 15(5):1110–1120

    Article  Google Scholar 

  42. Roitberg A, Pollert T, Haurilet M, Martin M, Stiefelhagen R (2019) Analysis of deep fusion strategies for multi-modal gesture recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 0–0

  43. Schoutens W (2000) The Askey scheme of orthogonal polynomials. In: Stochastic processes and orthogonal polynomials, Springer, pp 1–13

  44. Shanmuganathan V, Yesudhas HR, Khan MS, Khari M, Gandomi AH (2020) R-CNN and wavelet feature extraction for hand gesture recognition with EMG signals. Neural Comput Applic 32(21):16723–16736

    Article  Google Scholar 

  45. Singh C, Singh J (2018) Multi-channel versus quaternion orthogonal rotation invariant moments for color image representation. Digital Signal Process 78:376–392

    Article  MathSciNet  Google Scholar 

  46. Singh C, Singh J (2018) Quaternion generalized Chebyshev-Fourier and pseudo-Jacobi-Fourier moments for color object recognition. Optic Laser Technol 106:234–250

    Article  Google Scholar 

  47. Singh C, Upneja R (2014) Accurate calculation of high order pseudo-Zernike moments and their numerical stability. Digital Signal Process 27:95–106

    Article  Google Scholar 

  48. Suarez J, Murphy RR (2012) Hand gesture recognition with depth images: a review. In: 2012 IEEE RO-MAN: The 21st IEEE international symposium on robot and human interactive communication, IEEE, pp 411–417

  49. Sykora P, Kamencay P, Hudec R (2014) Comparison of SIFT and SURF methods for use on hand gesture recognition based on depth map. Aasri Procedia 9:19–24

    Article  Google Scholar 

  50. Teague MR (1980) Image analysis via the general theory of moments. JOSA 70(8):920–930

    Article  MathSciNet  Google Scholar 

  51. Wang C, Liu Z, Chan S-C (2014) Superpixel-based hand gesture recognition with kinect depth camera. IEEE Trans Multimed 17(1):29–39

    Article  Google Scholar 

  52. Yang T, Ma J, Miao Y, Wang X, Xiao B, He B, Meng Q (2019) Quaternion weighted spherical Bessel-Fourier moment and its invariant for color image reconstruction and object recognition. Inf Sci 505:388–405

    Article  MathSciNet  MATH  Google Scholar 

  53. Yap P-T, Paramesran R (2004) Jacobi moments as image features. In: 2004 IEEE Region 10 conference TENCON 2004, IEEE, pp 594–597

  54. Zhang F, Liu Y, Zou C, Wang Y (2018) Hand gesture recognition based on HOG-LBP feature. In: 2018 IEEE International instrumentation and measurement technology conference (i2MTC), IEEE, pp 1–6

  55. Zhu H, Liu M, Ji H, Li Y (2010) Combined invariants to blur and rotation using Zernike moment descriptors. Pattern Anal Applic 13(3):309–319

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgments

The authors thankfully acknowledge the Laboratory of Intelligent Systems and Applications (LSIA) for his support to achieve this work.

Funding

This work is supported by the Moroccan National Center for Scientific and Technical Research (CNRST) under the Excellence Research Scholarships Program.

Author information

Authors and Affiliations

  1. Laboratory of Intelligent Systems and Application (LSIA), Faculty of Sciences and Technology, University Sidi Mohamed Ben Abdellah, Fez, Morocco

    Ilham El Ouariachi, Rachid Benouini, Khalid Zenkouar & Arsalane Zarghili

  2. National School of Applied Sciences, University Sidi Mohamed Ben Abdellah, Fez, Morocco

    Hakim El Fadili

Authors
  1. Ilham El Ouariachi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rachid Benouini
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Khalid Zenkouar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Arsalane Zarghili
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hakim El Fadili
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ilham El Ouariachi.

Ethics declarations

Conflict of Interests

The authors declare no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

El Ouariachi, I., Benouini, R., Zenkouar, K. et al. RGB-D feature extraction method for hand gesture recognition based on a new fast and accurate multi-channel cartesian Jacobi moment invariants. Multimed Tools Appl 81, 12725–12757 (2022). https://doi.org/10.1007/s11042-022-12161-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12161-2

Keywords

Search

Navigation