Skip to main content
SpringerLink
Log in

Exploiting skeleton-based gait events with attention-guided residual deep learning model for human identification

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Human identification using unobtrusive visual features is a daunting task in smart environments. Gait is among adequate biometric features when the camera cannot correctly capture the human face due to environmental factors. In recent years, gait-based human identification using skeleton data has been intensively studied using a variety of feature extractors and more sophisticated deep learning models. Although skeleton data is susceptible to changes in covariate variables, resulting in noisy data, most existing algorithms employ a single feature extraction technique for all frames to generate frame-level feature maps. This results in degraded performance and additional features, necessitating increased computing power. This paper proposes a robust feature extractor that extracts a quantitative summary of gait event-specific information, thereby reducing the total number of features throughout the gait cycle. In addition, a novel Attention-guided LSTM-based deep learning model with residual connections is proposed to learn the extracted features for gait recognition. The proposed approach outperforms the state-of-the-art works on five publicly available datasets on various benchmark evaluation protocols and metrics. Further, the CMC test revealed that the proposed model obtained higher than 97% Accuracy in lower-level ranks on these datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data Availability

All the experiments are conducted utilizing publicly accessible datasets.

References

  1. Sepas-Moghaddam A, Etemad A (2023) Deep gait recognition: A survey. IEEE Trans Pattern Anal Mach Intell 45(1):264–284. https://doi.org/10.1109/TPAMI.2022.3151865

    Article  Google Scholar 

  2. Zhao A, Dong J, Li J et al (2022) Associated spatio-temporal capsule network for gait recognition. IEEE Trans Multimed 24:846–860. https://doi.org/10.1109/TMM.2021.3060280

    Article  Google Scholar 

  3. Ye M, Yang C, Stankovic V et al (2020) Distinct feature extraction for video-based gait phase classification. IEEE Trans Multimed 22(5):1113–1125. https://doi.org/10.1109/TMM.2019.2942479

    Article  Google Scholar 

  4. Liao R, Yu S, An W et al (2020) A model-based gait recognition method with body pose and human prior knowledge. Pattern Recognit 98:107069. https://doi.org/10.1016/j.patcog.2019.107069

    Article  Google Scholar 

  5. Fan C, Peng Y, Cao C, et al (2020) Gaitpart: Temporal part-based model for gait recognition. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 14213–14221. https://doi.org/10.1109/CVPR42600.2020.01423

  6. Wang X, Feng S, Yan WQ (2021) Human gait recognition based on self-adaptive hidden markov model. IEEE/ACM Trans Comput Biol Bioinf 18(3):963–972. https://doi.org/10.1109/TCBB.2019.2951146

    Article  Google Scholar 

  7. Xu W (2021) Graph-optimized coupled discriminant projections for cross-view gait recognition. Appl Intell 51(11):8149–8161. https://doi.org/10.1007/s10489-021-02322-5

    Article  Google Scholar 

  8. Webster JB, Darter BJ (2019) 4 - principles of normal and pathologic gait. In: Webster JB, Murphy DP (eds) Atlas of orthoses and assistive devices, 5th edn. Elsevier, Philadelphia, pp 49–62.e1. https://doi.org/10.1016/B978-0-323-48323-0.00004-4

  9. Li N, Zhao X (2022) A strong and robust skeleton-based gait recognition method with gait periodicity priors. IEEE Transactions on Multimedia pp 1–1. https://doi.org/10.1109/TMM.2022.3154609

  10. Han J, Bhanu B (2006) Individual recognition using gait energy image. IEEE Trans Pattern Anal Mach Intell 28(2):316–322. https://doi.org/10.1109/TPAMI.2006.38

    Article  Google Scholar 

  11. Huang X, Boulgouris NV (2012) Gait recognition with shifted energy image and structural feature extraction. IEEE Trans Image Process 21(4):2256–2268. https://doi.org/10.1109/TIP.2011.2180914

    Article  MathSciNet  MATH  Google Scholar 

  12. Liao R, Li Z, Bhattacharyya S et al (2022) Posemapgait: A model-based gait recognition method with pose estimation maps and graph convolutional networks. Neurocomputing, p 501. https://doi.org/10.1016/j.neucom.2022.06.048

  13. Zheng L, Zha Y, Kong D et al (2022) Multi-branch angle aware spatial temporal graph convolutional neural network for model-based gait recognition. IET Cyber-Systems and Robotics. https://doi.org/10.1049/csy2.12052

    Article  Google Scholar 

  14. Li X, Makihara Y, Xu C, et al (2020) End-to-end model-based gait recognition. In: Proceedings of the Asian Conference on Computer Vision (ACCV)

  15. Limcharoen P, Khamsemanan N, Nattee C (2021) Gait recognition and re-identification based on regional lstm for 2-second walks. IEEE Access 9:112057–112068. https://doi.org/10.1109/ACCESS.2021.3102936

    Article  Google Scholar 

  16. Luo J, Tjahjadi T (2020) View and clothing invariant gait recognition via 3d human semantic folding. IEEE Access 8:100365–100383. https://doi.org/10.1109/ACCESS.2020.2997814

    Article  Google Scholar 

  17. Zhang Y, Huang Y, Yu S et al (2020) Cross-view gait recognition by discriminative feature learning. IEEE Trans Image Process 29:1001–1015. https://doi.org/10.1109/TIP.2019.2926208

    Article  MathSciNet  MATH  Google Scholar 

  18. Rao H, Wang S, Hu X et al (2022) A self-supervised gait encoding approach with locality-awareness for 3d skeleton based person re-identification. IEEE Trans Pattern Anal Mach Intell 44(10):6649–6666. https://doi.org/10.1109/TPAMI.2021.3092833

    Article  Google Scholar 

  19. Han F, Reily B, Hoff W et al (2017) Space-time representation of people based on 3d skeletal data: A review. Comput Vis Image Understand 158:85–105. https://doi.org/10.1016/j.cviu.2017.01.011

    Article  Google Scholar 

  20. Khamsemanan N, Nattee C, Jianwattanapaisarn N (2018) Human identification from freestyle walks using posture-based gait feature. IEEE Trans Inf Forensics Secur 13(1):119–128. https://doi.org/10.1109/TIFS.2017.2738611

    Article  Google Scholar 

  21. Zhang Z (2012) Microsoft kinect sensor and its effect. IEEE MultiMed 19(2):4–10. https://doi.org/10.1109/MMUL.2012.24

    Article  Google Scholar 

  22. Bari ASMH, Gavrilova ML (2019) Artificial neural network based gait recognition using kinect sensor. IEEE Access 7:162708–162722. https://doi.org/10.1109/ACCESS.2019.2952065

    Article  Google Scholar 

  23. Zhong Y, Yan Q (2022) Spatio-temporal stacking model for skeleton-based action recognition. Appl Intell 52(11):12116–12130. https://doi.org/10.1007/s10489-021-02994-z

    Article  Google Scholar 

  24. Li X, Makihara Y, Xu C, et al (2020) Gait recognition via semi-supervised disentangled representation learning to identity and covariate features. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 13306–13316. https://doi.org/10.1109/CVPR42600.2020.01332

  25. Ma G, Wu L, Wang Y (2017) A general subspace ensemble learning framework via totally-corrective boosting and tensor-based and local patch-based extensions for gait recognition. Pattern Recognit 66:280–294. https://doi.org/10.1016/j.patcog.2017.01.003

    Article  Google Scholar 

  26. Han J, Bhanu B (2006) Individual recognition using gait energy image. IEEE Trans Pattern Anal Mach Intell 28(2):316–322. https://doi.org/10.1109/TPAMI.2006.38

    Article  Google Scholar 

  27. Elharrouss O, Almaadeed N, Al-Maadeed S et al (2021) Gait recognition for person re-identification. J Supercomput 77:3653–3672. https://doi.org/10.1007/s11227-020-03409-5

    Article  Google Scholar 

  28. Tang J, Luo J, Tjahjadi T et al (2017) Robust arbitrary-view gait recognition based on 3D partial similarity matching. IEEE Trans Image Process 26(1):7–22. https://doi.org/10.1109/TIP.2016.2612823

    Article  MathSciNet  MATH  Google Scholar 

  29. Sun J, Wang Y, Li J et al (2018) View-invariant gait recognition based on kinect skeleton feature. Multimed Tools Appl 77(19):24909–24935. https://doi.org/10.1007/s11042-018-5722-1

    Article  Google Scholar 

  30. Choi S, Kim J, Kim W et al (2019) Skeleton-based gait recognition via robust frame-level matching. IEEE Trans Inf Forensics Secur 14(10):2577–2592. https://doi.org/10.1109/TIFS.2019.2901823

    Article  Google Scholar 

  31. Hosni N, Amor BB (2020) A geometric convnet on 3d shape manifold for gait recognition. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 3725–3734. https://doi.org/10.1109/CVPRW50498.2020.00434

  32. Rashmi M, Guddeti RMR (2022) Human identification system using 3D skeleton-based gait features and LSTM model. J Vis Commun Image Repre 82:103416. https://doi.org/10.1016/j.jvcir.2021.103416

    Article  Google Scholar 

  33. Kastaniotis D, Theodorakopoulos I, Economou G et al (2016) Gait based recognition via fusing information from euclidean and riemannian manifolds. Pattern Recognit Lett 84:245–251. https://doi.org/10.1016/j.patrec.2016.10.012

    Article  Google Scholar 

  34. Huynh-The T, Hua CH, Tu NA et al (2020) Learning 3d spatiotemporal gait feature by convolutional network for person identification. Neurocomput 397:192–202. https://doi.org/10.1016/j.neucom.2020.02.048

    Article  Google Scholar 

  35. Liu Y, Jiang X, Sun T, et al (2019) 3d gait recognition based on a cnn-lstm network with the fusion of skegei and da features. In: 2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp 1–8. https://doi.org/10.1109/AVSS.2019.8909881

  36. Li G, Guo L, Zhang R et al (2023) Transgait: Multimodal-based gait recognition with set transformer. Appl Intell 53(2):1535–1547. https://doi.org/10.1007/s10489-022-03543-y

    Article  Google Scholar 

  37. Savitzky A, Golay MJ (1964) Smoothing and differentiation of data by simplified least squares procedures. Anal Chem 36(8):1627–1639. https://doi.org/10.1021/ac60214a047

    Article  Google Scholar 

  38. Nambiar A, Bernardino A, Nascimento JC, et al (2017) Context-aware person re-identification in the wild via fusion of gait and anthropometric features. In: 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), pp 973–980. https://doi.org/10.1109/FG.2017.121

  39. Nambiar A, Bernardino A, Nascimento J, et al (2017) Towards view-point invariant person re-identification via fusion of anthropometric and gait features from kinect measurements. pp 108–119. https://doi.org/10.5220/0006165301080119

  40. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  41. Cho K, van Merriënboer B, Gulcehre C, et al (2014) Learning phrase representations using RNN encoder–decoder for statistical machine translation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for computational linguistics, Doha, Qatar, pp 1724–1734. https://doi.org/10.3115/v1/D14-1179

  42. Vaswani A, Shazeer N, Parmar N et al (2017) Attention is all you need. In: Guyon I, Luxburg UV, Bengio S et al (eds) Advances in neural information processing systems, vol 30. Curran Associates Inc

    Google Scholar 

  43. Kastaniotis D, Theodorakopoulos I, Theoharatos C et al (2015) A framework for gait-based recognition using kinect. Pattern Recognit Lett 68:327–335. https://doi.org/10.1016/j.patrec.2015.06.020, special Issue on Soft Biometrics

  44. Kastaniotis D, Theodorakopoulos I, Economou G, et al (2013) Gait-based gender recognition using pose information for real time applications. In: 2013 18th International Conference on Digital Signal Processing (DSP), pp 1–6. https://doi.org/10.1109/ICDSP.2013.6622766

  45. Andersson VO, Araujo RM (2015) Person identification using anthropometric and gait data from kinect sensor. In: Proceedings of the 29th AAAI Conference on artificial intelligence. AAAI Press, AAAI’15, pp 425-431. https://doi.org/10.1609/aaai.v29i1.9212

  46. Munaro M, Ghidoni S, Dizmen DT, et al (2014) A feature-based approach to people re-identification using skeleton keypoints. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), pp 5644–5651. https://doi.org/10.1109/ICRA.2014.6907689

  47. Munaro M, Basso A, Fossati A, et al (2014) 3d reconstruction of freely moving persons for re-identification with a depth sensor. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), pp 4512–4519. https://doi.org/10.1109/ICRA.2014.6907518

  48. Nanni L, Munaro M, Ghidoni S et al (2016) Ensemble of different approaches for a reliable person re-identification system. Appl Comput Inf 12(2):142–153. https://doi.org/10.1016/j.aci.2015.02.002

    Article  Google Scholar 

  49. Ball A, Rye D, Ramos F, et al (2012) Unsupervised clustering of people from ‘skeleton’ data. In: 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp 225–226. https://doi.org/10.1145/2157689.2157767

  50. Preis J, Kessel M, Werner M et al (2012) Gait recognition with kinect. 1st international workshop on kinect in pervasive computing. New Castle, UK, pp 1–4

    Google Scholar 

  51. Yang K, Dou Y, Lv S et al (2016) Relative distance features for gait recognition with kinect. J Vis Commun Image Repre 39:209–217. https://doi.org/10.1016/j.jvcir.2016.05.020

    Article  Google Scholar 

  52. Bari AH, Gavrilova ML (2022) Kinectgaitnet: Kinect-based gait recognition using deep convolutional neural network. Sensors 22(7):2631. https://doi.org/10.3390/s22072631

  53. Haque A, Alahi A, Fei-Fei L (2016) Recurrent attention models for depth-based person identification. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 1229–1238. https://doi.org/10.1109/CVPR.2016.138

  54. Li J, Qi L, Zhao A, et al (2017) Dynamic long short-term memory network for skeleton-based gait recognition. In: 2017 IEEE SmartWorld, ubiquitous intelligence & computing, advanced & trusted computed, scalable computing & communications, cloud & big data computing, internet of people and smart city innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI), pp 1–6. https://doi.org/10.1109/UIC-ATC.2017.8397466

  55. Rao H, Wang S, Hu X, et al (2020) Self-supervised gait encoding with locality-aware attention for person re-identification. In: Bessiere C (ed) Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20. International Joint Conferences on Artificial Intelligence Organization, pp 898–905. https://doi.org/10.24963/ijcai.2020/125

  56. Chen Y, Xia S, Zhao J et al (2022) Adversarial learning-based skeleton synthesis with spatial-channel attention for robust gait recognition. Multimed Tools Appl 82:1–16. https://doi.org/10.1007/s11042-022-12665-x

    Article  Google Scholar 

  57. Semwal VB, Singha J, Sharma PK et al (2017) An optimized feature selection technique based on incremental feature analysis for bio-metric gait data classification. Multimed Tools Appl 76(22):24457–24475. https://doi.org/10.1007/s11042-016-4110-y

    Article  Google Scholar 

  58. Justus D, Brennan J, Bonner S, et al (2018) Predicting the computational cost of deep learning models. In: 2018 IEEE International conference on big data (Big Data), pp 3873–3882. https://doi.org/10.1109/BigData.2018.8622396

  59. Tsironi E, Barros P, Weber C et al (2017) An analysis of convolutional long short-term memory recurrent neural networks for gesture recognition. Neurocomput 268:76–86. https://doi.org/10.1016/j.neucom.2016.12.088

    Article  Google Scholar 

Download references

Funding

This work did not receive any funding.

Author information

Authors and Affiliations

  1. Department of Information Technology, National Institute of Technology Karnataka, Surathkal, Mangalore, 575025, Karnataka, India

    Rashmi M & Ram Mohana Reddy Guddeti

Authors
  1. Rashmi M
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ram Mohana Reddy Guddeti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rashmi M.

Ethics declarations

Conflicts of interest

There are no conflicts of interest to declare by the authors.

Ethical standard

Not Applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

M, R., Guddeti, R.M.R. Exploiting skeleton-based gait events with attention-guided residual deep learning model for human identification. Appl Intell 53, 28711–28729 (2023). https://doi.org/10.1007/s10489-023-05019-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-05019-z

Keywords

Search

Navigation