Skip to main content
SpringerLink
Log in

A real-time recognition gait framework for personal authentication via image-based neural network: accelerated by feature reduction in time and frequency domains

  • Research
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

In recent years, personal authentication based on attitude estimation—gait recognition authentication has become a popular research topic because of its long-range, non-invasive, non-contact, high-precision, and other advantages. However, at present, most relevant research prefers to use the acquired original data directly for iteration and learning. As a result, it takes too long to learn relevant models in the use scenarios with complicated data and heavy human traffic, such as airports and railway stations, where real-time identification cannot be completed while maintaining accuracy, and thus a scheme to improve the learning and recognition speed is needed. Therefore, in this paper, we proposed an innovative real-time MediaPipe-based gait analysis framework and a new Composite Filter Feature Selection (CFFS) method via key nodes, angles, and lengths calculating. Then, based on the proposed method, we extract the aimed features as a new dataset and verified it by 1D-CNN neural network. Furthermore, we also applied Hilbert–Huang transform to investigate these extracted gait features in the frequency domain, improving the performance of our proposed framework to achieve real time under higher recognition accuracy. The experimental results show that the innovative gait recognition framework and data processing technology can reduce the gait feature data, speed up the process of gait recognition, and still maintain the original recognition accuracy. It can also be applied to various large, enclosed spaces with the huge human flow, which has played a role in improving the safety factor, saving labor costs, and accelerating economic consumption.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data availability

The datasets generated and analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. Alsaadi, I.M.: Study on most popular behavioral biometrics, advantages, disadvantages and recent applications: a review. Int. J. Sci. Technol. Res 10, 15–21 (2021)

    Google Scholar 

  2. Yang, W., Wang, S., Hu, J., Zheng, G., Valli, C.: Security and accuracy of fingerprint-based biometrics: a review. Symmetry 11(2), 141 (2019). https://doi.org/10.3390/sym11020141

    Article  Google Scholar 

  3. Huang, X., Nishimura, S., Wu, B.: A pose detection based continuous authentication system design via gait feature analysis. In: 2022 IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress, pp. 1–5 (2022)

  4. Prakash, Chandra, Rajesh, K., Namita, M.: Recent developments in human gait research: parameters, approaches, applications, machine learning techniques, datasets and challenges. Artif. Intell. Rev. 49, 1–40 (2018)

  5. Singh, J.P., Jain, S., Arora, S., et al.: Vision-based gait recognition: a survey. IEEE Access 6, 70497–70527 (2018)

    Article  Google Scholar 

  6. Yang, G., Tan, W., Jin, H., Zhao, T., Tu, L.: Review wearable sensing system for gait recognition. Clust. Comput. 22, 3021–3029 (2019)

    Article  Google Scholar 

  7. Wu, B., Wu, Y., Dong, R., et al.: Behavioral analysis of mowing workers based on hilbert-huang transform: an auxiliary movement analysis of manual mowing on the slopes of terraced rice fields. Agriculture 13(2), 489 (2023)

    Article  Google Scholar 

  8. Wu, B., Wu, Y., Nishimura, S., Jin, Q.: Analysis on the subdivision of skilled mowing movements on slopes. Sensors 22(4), 1372 (2022)

  9. Wu, B., Zhu, Y., Yu, K., Nishimura, S., Jin, Q.: The effect of eye movements and culture on product color selection. Hum. Centric Comput. Inform. Sci. 10(48) (2020)

  10. Wang, Z., Wu, B., Sato, K.: A depth camera-based warning system design for social distancing detection. In: 2021 IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress, pp. 901–906 (2021)

  11. Nordin, M.J., Saadoon, A.: A survey of gait recognition based on skeleton model for human identification. Res. J. Appl. Sci. Eng. Technol. 12(7), 756–763 (2016) https://doi.org/10.19026/rjaset.12.2751

  12. Wan, C., Wang, L., Phoha, V.V. (eds.): A survey on gait recognition. ACM Comput. Surv. (CSUR), 51(5), 1–35 (2018)

  13. Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Grundmann, M.: MediaPipe: a framework for building perception pipelines (2019). arXiv preprint arXiv:1906.08172.

  14. Holden, D., Saito, J., Komura, T.: A deep learning framework for character motion synthesis and editing. ACM Trans. Graph. (TOG) 35(4), 1–11 (2016)

    Article  Google Scholar 

  15. Holden, D., Saito, J., Komura, T., Joyce, T.: Learning motion manifolds with convolutional autoencoders. In: SIGGRAPH Asia 2015 technical briefs, pp. 1–4 (2015)

  16. Chang, Q., Maruyama, T.: Real-time stereo vision system: a multi-block matching on GPU. IEEE Access 6, 42030–42046 (2018)

    Article  Google Scholar 

  17. Kim, C.L., Kim, B.G.: Few-shot learning for facial expression recognition: a comprehensive survey. J. Real-Time Image Proc. 20, 52 (2023). https://doi.org/10.1007/s11554-023-01310-x

    Article  Google Scholar 

  18. Khan, M.A., Menouar, H., Hamila, R.: LCDnet: a lightweight crowd density estimation model for real-time video surveillance. J. Real Time Image Proc. 20, 29 (2023). https://doi.org/10.1007/s11554-023-01286-8

    Article  Google Scholar 

  19. Dong, R., Chang, Q., Ikuno, S.: A deep learning framework for realistic robot motion generation. Neural Comput. Appl. 1–14 (2021)

  20. Dong, R., Chen, Y., Cai, D., Nakagawa, S., Higaki, T., Asai, N.: Robot motion design using bunraku emotional expressions–focusing on Jo-Ha-Kyū in sounds and movements. Adv. Robot. 34(5), 299–312 (2020)

    Article  Google Scholar 

  21. Arshad, H., Khan, M.A., Sharif, M.I., Yasmin, M., Tavares, J.M.R., Zhang, Y.D., Satapathy, S.C.: A multilevel paradigm for deep convolutional neural network features selection with an application to human gait recognition. Expert. Syst. 39(7), e12541 (2022)

    Article  Google Scholar 

  22. Filipi Gonçalves dos Santos, C., Oliveira, D. D. S., A. Passos, L., Gonçalves Pires, R., Felipe Silva Santos, D., Pascotti Valem, L., Colombo, D.: Gait recognition based on deep learning: a survey. ACM Comput. Surv. (CSUR), 55(2), 1–34 (2022)

  23. Kong, Q., Wu, Z., Deng, Z., Klinkigt, M., Tong, B., Murakami, T.: Mmact: a large-scale dataset for cross modal human action understanding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8658–8667 (2019).

  24. Nixon, M.S., Carter, J.N.: Automatic recognition by gait. Proc. IEEE 94(11), 2013–2024 (2006)

    Article  Google Scholar 

  25. Liao, R., Yu, S., An, W., Huang, Y.: A model-based gait recognition method with body pose and human prior knowledge. Pattern Recogn. 98, 107069 (2020)

    Article  Google Scholar 

  26. Connor, P., Ross, A.: Biometric recognition by gait: a survey of modalities and features. Comput. Vis. Image Underst. 167, 1–27 (2018)

    Article  Google Scholar 

  27. Moeslund, T.B., Hilton, A., Krüger, V.: A survey of advances in vision-based human motion capture and analysis. Comput. Vis. Image Underst. 104(2–3), 90–126 (2006)

    Article  Google Scholar 

  28. Borges, P.V.K., Conci, N., Cavallaro, A.: Video-based human behavior understanding: a survey. IEEE Trans. Circ. Syst. Video Technol. 23(11), 1993–2008 (2013)

    Article  Google Scholar 

  29. Zhang, F., Bazarevsky, V., Vakunov, A., Tkachenka, A., Sung, G., Chang, C. L., Grundmann, M.: MediaPipe hands: on-device real-time hand tracking. arXiv preprint arXiv:2006 (2020)

  30. Ghanbari, S., Ashtyani, Z. P., & Masouleh, M. T.: User identification based on hand geometrical biometrics using media-pipe. In 2022 30th International Conference on Electrical Engineering (ICEE) pp. 373–378. (2022)

  31. Garg, S., Saxena, A., Gupta, R.: Yoga pose classification: a CNN and MediaPipe inspired deep learning approach for real-world application. J. Ambient Intell. Hum. Comput, 1–12 (2022)

  32. Castro, F.M., Marin-Jimenez, M.J., Guil, N., Pérez de la Blanca, N.: Multimodal feature fusion for CNN-based gait recognition: an empirical comparison. Neural Comput. Appl. 32, 14173–14193 (2020)

  33. Tang, W., Long, G., Liu, L., Zhou, T., Jiang, J., Blumenstein, M.: Rethinking 1d-cnn for time series classification: a stronger baseline. arXiv preprint arXiv:10061, 1–7 (2002)

  34. Wang, K., Ma, C., Qiao, Y., Lu, X., Hao, W., Dong, S.: A hybrid deep learning model with 1DCNN-LSTM-attention networks for short-term traffic flow prediction. Physica A 583, 126293 (2021)

    Article  Google Scholar 

  35. Chakraborty, J., Nandy, A.: Discrete wavelet transform based data representation in deep neural network for gait abnormality detection. Biomed. Signal Process. Control 62, 102076 (2020)

    Article  Google Scholar 

  36. Huang, N. E.: Hilbert–Huang transform and its applications. World Scientific (2014)

  37. Bracewell, R.N., Bracewell, R.N.: The fourier transform and its applications. McGraw-Hill, New York (1986)

    MATH  Google Scholar 

  38. Huang, N.E., Shen, Z., Long, S.R., Wu, M.C., Shih, H.H., Zheng, Q., Yen, N.-C., Tung, C., Liu, H.H.: The empirical mode decomposition and the Hilbert spectrum for non-linear and non-stationary time series analysis. Proc. R. Soc. Lond. Ser. A: Math. Phys. Eng. Sci. 454(1971), 903–995 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  39. Dong, R., Dongsheng, C., Soichiro, I.: Motion capture data analysis in the instantaneous frequency-domain using Hilbert–Huang transform. Sensors 20(22), 6534 (2020)

    Article  Google Scholar 

  40. Wu, B., Zhu, Y., Dong, R., Sato, K., Ikuno, S., Nishimura, S., Jin, Q.: Pre-braking behaviors analysis based on Hilbert–Huang transform. CCF Trans. Pervas. Comp. Interact (2022). https://doi.org/10.1007/s42486-022-00123-4

    Article  Google Scholar 

  41. Wu, B., Wu, Y., Dong, R., Sato, K., Ikuno, S., Nishimura, S., Jin, Q.: Behavioral analysis of mowing workers based on hilbert-huang transform: an auxiliary movement analysis of manual mowing on the slopes of terraced rice fields. Agriculture 13(2), 489 (2023)

    Article  Google Scholar 

  42. Kong, Q., Wu, Z., Deng, Z., Klinkigt, M., Tong, B., Murakami, T.: Mmact: A large-scale dataset for cross modal human action understanding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 8658–8667 (2019)

  43. Hansen, J.B., Kristiansen, N.H.: A data-based parametric biomechanical. Biomed. Eng. 13, 171–183 (2022)

    Google Scholar 

  44. Ramirez, H., Velastin, S.A., Aguayo, P., Fabregas, E., Farias, G.: Human activity recognition by sequences of skeleton features. Sensors 22, 3991 (2022). https://doi.org/10.3390/s22113991

    Article  Google Scholar 

  45. Matteo, M., Stefano, G., Deniz, T. D., Emanuele, M.: A feature-based approach to people re-identification using skeleton keypoints. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China, pp. 5644–5651 (2014). https://doi.org/10.1109/ICRA.2014.6907689

  46. ur Rehman, N., Mandic, D.P.: Multivariate empirical mode decomposition. Proc. R. Soc. A: Math. Phys. Eng. Sci. 466(2117), 1291–1302 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  47. ur Rehman, N., Park, C., Huang, N.E., Mandic, D.P.: EMD via MEMD: multivariate noise-aided computation of standard EMD. Adv. Adapt. Data Anal. 5(02), 1350007 (2013)

    Article  MathSciNet  Google Scholar 

  48. Dong, R., Ni, S., Ikuno, S.: Non-linear frequency analysis of COVID-19 spread in Tokyo using empirical mode decomposition. Sci. Rep. 12(1), 1–12 (2022)

    Google Scholar 

  49. Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Adv. Neural. Inf. Process. Syst. 33, 6840–6851 (2020)

    Google Scholar 

Download references

Acknowledgements

This research was funded by JSPS KAKENHI [Grant numbers JP21K11876, JP21K17833]. The authors wish to thank all the workers who participated in the experiments. Co-first author: Ran Dong; Corresponding author: Bo Wu. On behalf of all authors, the corresponding author states that there is no conflict of interest.

Author information

Authors and Affiliations

  1. Metaverse Research Institute, Waseda University, Tokorozawa, Saitama, 359-1192, Japan

    Xuan Huang

  2. School of Engineering, Chukyo University, Toyota, Aichi, 470-0393, Japan

    Ran Dong

  3. School of Computer Science, Tokyo University of Technology, Hachioji, Tokyo, 192-0982, Japan

    Bo Wu, Kiminori Sato & Soichiro Ikuno

  4. Faculty of Human Sciences, Waseda University, Tokorozawa, Saitama, 359-1192, Japan

    Zijun Wang & Shoji Nishimura

Authors
  1. Xuan Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ran Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bo Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kiminori Sato
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Soichiro Ikuno
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zijun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Shoji Nishimura
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization, XH,RD and BW; methodology, XH,RD and BW; software, XH and RD; validation, XH and RD; formal analysis, XH and BW; investigation, XH and BW; resources, RD, BW, KS, SI and SN; data curation, XH and RD; writing—original draft preparation, XH,RD and BW; writing—review and editing, RD, BW, KS, SI, ZW and SN; visualization, XH,RD and BW; supervision, BW; project administration, RD and BW; funding acquisition, RD and BW; All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Bo Wu.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, X., Dong, R., Wu, B. et al. A real-time recognition gait framework for personal authentication via image-based neural network: accelerated by feature reduction in time and frequency domains. J Real-Time Image Proc 20, 92 (2023). https://doi.org/10.1007/s11554-023-01349-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11554-023-01349-w

Keywords

Search

Navigation