Advertisement

Applying adversarial auto-encoder for estimating human walking gait abnormality index

  • Trong-Nguyen NguyenEmail author
  • Jean Meunier
Industrial and commercial application
  • 2 Downloads

Abstract

This paper proposes an approach that estimates a human walking gait abnormality index using an adversarial auto-encoder (AAE), i.e., a combination of auto-encoder and generative adversarial network (GAN). Since most GAN-based models have been employed as data generators, our work introduces another perspective of their application. This method directly works on a sequence of 3D point clouds representing the walking postures of a subject. By fitting a cylinder onto each point cloud and feeding cylindrical histograms to an appropriate AAE, our system is able to provide different measures that may be used as gait abnormality indices. The combinations of such quantities are also investigated to obtain improved indicators. The ability of our method is demonstrated by experimenting on a large dataset of nearly 100 thousands point clouds, and the results outperform related approaches that employ different input data types.

Keywords

Gait Adversarial Auto-encoder Point cloud Posture Depth camera Kinect Mirror 

Notes

Acknowledgements

The authors would like to thank the NSERC (Natural Sciences and Engineering Research Council of Canada) for supporting this work (Discovery Grant RGPIN-2015-05671). We also thank Hoang Anh Nguyen (Airspace Systems Inc., CA, USA) for useful discussions.

References

  1. 1.
    Auvinet E, Meunier J, Multon F (2012) Multiple depth cameras calibration and body volume reconstruction for gait analysis. In: 2012 11th international conference on information science, signal processing and their applications (ISSPA), pp 478–483.  https://doi.org/10.1109/ISSPA.2012.6310598
  2. 2.
    Auvinet E, Multon F, Meunier J (2015) New lower-limb gait asymmetry indices based on a depth camera. Sensors 15(3):4605–4623.  https://doi.org/10.3390/s150304605 CrossRefGoogle Scholar
  3. 3.
    Bauckhage C, Tsotsos JK, Bunn FE (2009) Automatic detection of abnormal gait. Image Vis Comput 27(1):108–115CrossRefGoogle Scholar
  4. 4.
    Bei S, Zhen Z, Xing Z, Taocheng L, Qin L (2018) Movement disorder detection via adaptively fused gait analysis based on Kinect sensors. IEEE Sens J 18(17):7305–7314.  https://doi.org/10.1109/JSEN.2018.2839732 CrossRefGoogle Scholar
  5. 5.
    Bigy AAM, Banitsas K, Badii A, Cosmas J (2015) Recognition of postures and freezing of gait in Parkinson’s disease patients using microsoft Kinect sensor. In: 2015 7th international IEEE/EMBS conference on neural engineering (NER), pp 731–734.  https://doi.org/10.1109/NER.2015.7146727
  6. 6.
    Chaaraoui AA, Padilla-López JR, Flórez-Revuelta F (2015) Abnormal gait detection with RGB-D devices using joint motion history features. In: 2015 11th IEEE international conference and workshops on automatic face and gesture recognition (FG). vol 7. IEEE, pp 1–6Google Scholar
  7. 7.
    Davis JW (2001) Hierarchical motion history images for recognizing human motion. In: IEEE workshop on detection and recognition of events in video. Proceedings. IEEE, pp 39–46Google Scholar
  8. 8.
    Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Ghahramani Z, Welling M, Cortes C, Lawrence ND, Weinberger KQ (eds) Advances in neural information processing systems 27, Curran Associates, Inc., pp 2672–2680. http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf
  9. 9.
    Han J, Bhanu B (2006) Individual recognition using gait energy image. IEEE Trans Pattern Anal Mach Intell 28(2):316–322.  https://doi.org/10.1109/TPAMI.2006.38 CrossRefGoogle Scholar
  10. 10.
    Huang NE, Shen Z, Long SR, Wu MC, Shih HH, Zheng Q, Yen NC, Tung CC, Liu HH (1998) The empirical mode decomposition and the hilbert spectrum for nonlinear and non-stationary time series analysis. Proc R Soc Lond A Math Phys Eng Sci 454(1971):903–995.  https://doi.org/10.1098/rspa.1998.0193 MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Jiang S, Wang Y, Zhang Y, Sun J (2015) Real time gait recognition system based on Kinect skeleton feature. Springer International Publishing, Cham, pp 46–57.  https://doi.org/10.1007/978-3-319-16628-5_4 Google Scholar
  12. 12.
    Kim H, Kim Y, Ko D, Kim J, Lee EC (2014) Pointing gesture interface for large display environments based on the kinect skeleton model. In: Park JJJH, Pan Y, Kim CS, Yang Y (eds) Future information technology. Springer, Berlin, pp 509–514CrossRefGoogle Scholar
  13. 13.
    Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:14126980
  14. 14.
    López-Fernández D, Madrid-Cuevas F, Carmona-Poyato A, Munoz-Salinas R, Medina-Carnicer R (2016) A new approach for multi-view gait recognition on unconstrained paths. J Vis Commun Image Represent 38:396–406.  https://doi.org/10.1016/j.jvcir.2016.03.020 CrossRefGoogle Scholar
  15. 15.
    Makhzani A, Shlens J, Jaitly N, Goodfellow I (2016) Adversarial autoencoders. In: International conference on learning representations. arXiv:1511.05644
  16. 16.
    Martinelli M, Tronci E, Dipoppa G, Balducelli C (2004) Electric power system anomaly detection using neural networks. In: Negoita MG, Howlett RJ, Jain LC (eds) Knowledge-based intelligent information and engineering systems. Springer, Berlin, pp 1242–1248CrossRefGoogle Scholar
  17. 17.
    Nguyen TN, Meunier J (2018) Walking gait dataset: point clouds, skeletons and silhouettes. Technical Report 1379, DIRO, University of Montreal. http://www.iro.umontreal.ca/~labimage/GaitDataset/dataset.pdf
  18. 18.
    Nguyen TN, Huynh HH, Meunier J (2014) Extracting silhouette-based characteristics for human gait analysis using one camera. In: Proceedings of the 5th symposium on information and communication technology, ACM, New York, NY, USA, SoICT ’14, pp 171–177.  https://doi.org/10.1145/2676585.2676612
  19. 19.
    Nguyen TN, Huynh HH, Meunier J (2016) Skeleton-based abnormal gait detection. Sensors 16(11):1792.  https://doi.org/10.3390/s16111792 CrossRefGoogle Scholar
  20. 20.
    Nguyen TN, Huynh HH, Meunier J (2018a) 3d reconstruction with time-of-flight depth camera and multiple mirrors. IEEE Access 6:38106–38114.  https://doi.org/10.1109/ACCESS.2018.2854262 CrossRefGoogle Scholar
  21. 21.
    Nguyen TN, Huynh HH, Meunier J (2018b) Assessment of gait normality using a depth camera and mirrors. In: 2018 IEEE EMBS international conference on biomedical health informatics (BHI), Las Vegas, NV, USA, pp 37–41.  https://doi.org/10.1109/BHI.2018.8333364
  22. 22.
    Nguyen TN, Huynh HH, Meunier J (2018c) Using ToF camera and two mirrors for 3D reconstruction of dynamic objects. Technical Report 1380, DIRO, University of Montreal. http://www.iro.umontreal.ca/~labimage/GaitDataset/reconstruct3D.pdf
  23. 23.
    Prabhu P, Karunakar A, Anitha H, Pradhan N (2018) Classification of gait signals into different neurodegenerative diseases using statistical analysis and recurrence quantification analysis. Pattern Recogn Lett.  https://doi.org/10.1016/j.patrec.2018.05.006 Google Scholar
  24. 24.
    Ren P, Tang S, Fang F, Luo L, Xu L, Bringas-Vega ML, Yao D, Kendrick KM, Valdes-Sosa PA (2017) Gait rhythm fluctuation analysis for neurodegenerative diseases by empirical mode decomposition. IEEE Trans Biomed Eng 64(1):52–60.  https://doi.org/10.1109/TBME.2016.2536438 CrossRefGoogle Scholar
  25. 25.
    Rodriguez S, Pérez K, Quintero C, López J, Rojas E, Calderón J (2016) Identification of multimodal human–robot interaction using combined kernels. In: Snášel V, Abraham A, Krömer P, Pant M, Muda AK (eds) Innovations in bio-inspired computing and applications. Springer International Publishing, Cham, pp 263–273CrossRefGoogle Scholar
  26. 26.
    Roth K, Lucchi A, Nowozin S, Hofmann T (2017) Stabilizing training of generative adversarial networks through regularization. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R (eds) Advances in neural information processing systems 30. Curran Associates Inc, Red Hook, pp 2018–2028Google Scholar
  27. 27.
    Sakurada M, Yairi T (2014) Anomaly detection using autoencoders with nonlinear dimensionality reduction. In: Proceedings of the MLSDA 2014 2nd workshop on machine learning for sensory data analysis, ACM, New York, NY, USA, MLSDA’14, pp 4:4–4:11.  https://doi.org/10.1145/2689746.2689747
  28. 28.
    Shotton J, Fitzgibbon A, Cook M, Sharp T, Finocchio M, Moore R, Kipman A, Blake A (2011) Real-time human pose recognition in parts from single depth images. CVPR 2011:1297–1304.  https://doi.org/10.1109/CVPR.2011.5995316 Google Scholar
  29. 29.
    Shotton J, Girshick R, Fitzgibbon A, Sharp T, Cook M, Finocchio M, Moore R, Kohli P, Criminisi A, Kipman A, Blake A (2013) Efficient human pose estimation from single depth images. IEEE Trans Pattern Anal Mach Intell 35(12):2821–2840.  https://doi.org/10.1109/TPAMI.2012.241 CrossRefGoogle Scholar
  30. 30.
    Webber CL Jr, Marwan N (eds) (2015) Recurrence quantification analysis: theory and best practices. Springer, Cham.  https://doi.org/10.1007/978-3-319-07155-8
  31. 31.
    Wilson AC, Roelofs R, Stern M, Srebro N, Recht B (2017) The marginal value of adaptive gradient methods in machine learning. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R (eds) Advances in neural information processing systems 30. Curran Associates Inc, Red Hook, pp 4148–4158Google Scholar
  32. 32.
    Yang Y, Liu R, Deng C, Gao X (2016) Multi-task human action recognition via exploring super-category. Signal Process 124:36–44.  https://doi.org/10.1016/j.sigpro.2015.10.035 CrossRefGoogle Scholar
  33. 33.
    Yang Y, Deng C, Gao S, Liu W, Tao D, Gao X (2017a) Discriminative multi-instance multitask learning for 3d action recognition. IEEE Trans Multimed 19(3):519–529.  https://doi.org/10.1109/TMM.2016.2626959 CrossRefGoogle Scholar
  34. 34.
    Yang Y, Deng C, Tao D, Zhang S, Liu W, Gao X (2017b) Latent max-margin multitask learning with skelets for 3-d action recognition. IEEE Trans Cybern 47(2):439–448.  https://doi.org/10.1109/TCYB.2016.2519448 Google Scholar
  35. 35.
    Yu TH, Kim TK, Cipolla R (2010) Real-time action recognition by spatiotemporal semantic and structural forest. In: Proceedings of the British machine vision conference, BMVA Press, pp 52.1–52.12.  https://doi.org/10.5244/C.24.52

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2019

Authors and Affiliations

  1. 1.Image Processing Laboratory, DIROUniversity of MontrealMontrealCanada

Personalised recommendations