Skip to main content

Advertisement

SpringerLink
Log in

Compressive sensing based recognition of human upper limb motions with kinect skeletal data

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Limb motion recognition and assessment tasks have become one of the most substantial techniques in clinical environment to evaluate the motor capabilities of neurological patients. However, most of the recognition task are often time consuming and computationally expensive as well. In this paper, we propose a compressive sensing based upper limb motion recognition method using kinect skeletal data. Participants with no form of disabilities were selected for this study. Participants were requested to perform three limb actions and their upper limb motions are captured using kinect sensor. The skeletal data from the kinect sensor chosen for the recognition task is initially converted into RGB image. To alleviate the storage and computational issues, images from skeletal data are further compressed using compressive sensing paradigm. The compressed images are then fed onto a feed forward convolutional neural network for learning and recognizing actions. The recognition task is thus performed directly in the compressed domain avoiding the need for the usage of expensive decompression algorithms. Our proposed method is tested on the acquired data as well as two public benchmark datasets namely MSR Action 3D dataset and KARD dataset. An average recognition accuracy of 96.82%, 94.88% and 97.22% is obtained with our proposed method when tested on acquired dataset, MSR Action 3D dataset and KARD dataset respectively. The proposed method is also compared with several state of the art methods available in literature. Though there is not much increase in the average recognition accuracy of the proposed method on KARD dataset, an increase in average recognition accuracy of around 0.03% to 7.38% is achieved with the proposed method on MSR Action 3D dataset compared to many state of the art methods available in literature.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. MSRAction3D dataset <https://www.uow.edu.au/~wanqing/#Datasets>

  2. KARD dataset <https://data.mendeley.com/datasets/k28dtm7tr6/1>

References

  1. Akkaladevi SC, Heindl C (2015) Action recognition for human robot interaction in industrial applications. 2015 IEEE Int Conf Comput Graph Vis Inf Secur CGVIS 2016; 94–99.

  2. Asaeda M, Kuwahara W, Fujita N, Yamasaki T, Adachi N (2018) Validity of motion analysis using the Kinect system to evaluate single leg stance in patients with hip disorders. Gait Posture 62:458–462

    Article  Google Scholar 

  3. Ashwini K, Amutha R (2018) Compressive sensing based simultaneous fusion and compression of multi-focus images using learned dictionary. Multimed Tools Appl 77:25889–25904

    Article  Google Scholar 

  4. Ashwini K, Amutha R (2018) Fast and secured cloud assisted recovery scheme for compressively sensed signals using new chaotic system. Multimed Tools Appl 77:31581–31606

    Article  Google Scholar 

  5. Atrsaei A, Salarieh H, Alasty A (2016) Human arm motion tracking by orientation-based fusion of inertial sensors and Kinect using unscented Kalman filter. J Biomech Eng 138:091005

    Article  Google Scholar 

  6. Biswas D, Cranny A, Gupta N, Maharatna K, Achner J, Klemke J, Jöbges M, Ortmann S (2015) Recognizing upper limb movements with wrist worn inertial sensors using k-means clustering classification. Hum Mov Sci 40:59–76

    Article  Google Scholar 

  7. Bobin J, Starck JL, Ottensamer R (2008) Compressed sensing in astronomy. IEEE J Sel Top Signal Process 2:718–726

    Article  Google Scholar 

  8. Candès E (2009) Compressive sampling. Proc Int Congr Math Madrid, August 22–30(2006):1433–1452

    MATH  Google Scholar 

  9. Chen C, Liu K, Kehtarnavaz N (2016) Real-time human action recognition based on depth motion maps. J Real-Time Image Process 12:155–163

    Article  Google Scholar 

  10. Chen D, Cai Y, Cui J et al (2018) Risk factors identification and visualization for work-related musculoskeletal disorders with wearable and connected gait analytics system and kinect skeleton models. Smart Heal 7–8:60–77

    Article  Google Scholar 

  11. Cippitelli E, Gambi E, Spinsante S et al (2016) Evaluation of a skeleton-based method for human activity recognition on a large-scale RGB-D dataset. 14(6)

  12. Da Poian G, Bernardini R, Rinaldo R (2015) Separation and analysis of fetal-ECG signals from compressed sensed abdominal ECG recordings. IEEE Trans Biomed Eng 63(6):1269–1279

    Article  Google Scholar 

  13. Dhiman C, Vishwakarma DK (2019) A robust framework for abnormal human action recognition using R transform and zernike moments in depth videos. IEEE Sensors J 19:5195–5203

    Article  Google Scholar 

  14. Ding W, Liu K, Belyaev E, Cheng F (2018) Tensor-based linear dynamical systems for action recognition from 3D skeletons. Pattern Recogn 77:75–86

    Article  Google Scholar 

  15. Elmadany NED, He Y, Guan L (2018) Information fusion for human action recognition via Biset/multiset Globality locality preserving canonical correlation analysis. IEEE Trans Image Process 27:5275–5287

    Article  MathSciNet  Google Scholar 

  16. Evangelidis G, Singh G, Horaud R (2014) Skeletal quads: human action recognition using joint quadruples. Proc - Int Conf Pattern Recognit:4513–4518

  17. Gaglio S, Lo Re G, Morana M (2015) Human activity recognition process using 3-D posture data. IEEE Trans Human-Machine Syst 45:586–597

    Article  Google Scholar 

  18. Gharaee Z (2020) Hierarchical growing grid networks for skeleton based action recognition. Cogn Syst Res 63:11–29

    Article  Google Scholar 

  19. Guerra J, Uddin J, Nilsen D et al (2017) Capture, learning, and classification of upper extremity movement primitives in healthy controls and stroke patients. IEEE Int Conf Rehabil Robot:547–554

  20. Huang M, Cai GR, Zhang HB, Yu S, Gong DY, Cao DL, Li S, Su SZ (2018) Discriminative parts learning for 3D human action recognition. Neurocomputing 291:84–96

    Article  Google Scholar 

  21. Ijjina EP, Mohan CK (2014) Human action recognition based on mocap information using convolution neural networks. Proc - 2014 13th Int Conf Mach Learn Appl ICMLA 2014:159–164

    Google Scholar 

  22. Jansi R, Amutha R (2018) A novel chaotic map based compressive classification scheme for human activity recognition using a tri-axial accelerometer. Multimed Tools Appl 77:31261–31280

    Article  Google Scholar 

  23. Ji X, Cheng J, Feng W, Tao D (2018) Skeleton embedded motion body partition for human action recognition using depth sequences. Signal Process 143:56–68

    Article  Google Scholar 

  24. Khaire P, Kumar P, Imran J (2018) Combining CNN streams of RGB-D and skeletal data for human activity recognition. Pattern Recogn Lett 115:107–116

    Article  Google Scholar 

  25. Kiran Kumar E, Kishore PVV, Anil Kumar D, et al. (2018) Early estimation model for 3D-discrete indian sign language recognition using graph matching. J King Saud Univ - Comput Inf Sci. Epub ahead of print. https://doi.org/10.1016/j.jksuci.2018.06.008.

  26. Kumar P, Gauba H, Pratim Roy P, Prosad Dogra D (2017) A multimodal framework for sensor based sign language recognition. Neurocomputing 259:21–38

    Article  Google Scholar 

  27. Kumar P, Gauba H, Roy PP, Dogra DP (2017) Coupled HMM-based multi-sensor data fusion for sign language recognition. Pattern Recogn Lett 86:1–8

    Article  Google Scholar 

  28. Lee SI, Adans-Dester CP, Grimaldi M, Dowling AV, Horak PC, Black-Schaffer RM, Bonato P, Gwin JT (2018) Enabling stroke rehabilitation in home and community settings: a wearable sensor-based approach for upper-limb motor training. IEEE J Transl Eng Heal Med 6:1–11

    Article  Google Scholar 

  29. Li G, Li C (2020) Learning skeleton information for human action analysis using Kinect. Signal Process Image Commun 84:115814

    Article  Google Scholar 

  30. Li Q, Lin W, Li J (2018) Human activity recognition using dynamic representation and matching of skeleton feature sequences from RGB-D images. Signal Process Image Commun 68:265–272

    Article  Google Scholar 

  31. Liu J, Wang Z, Liu H (2020) HDS-SP: a novel descriptor for skeleton-based human action recognition. Neurocomputing 385:22–32

    Article  Google Scholar 

  32. Mazomenos EB, Biswas D, Cranny A, Rajan A, Maharatna K, Achner J, Klemke J, Jobges M, Ortmann S, Langendorfer P (2016) Detecting elementary arm movements by tracking upper limb joint angles with MARG sensors. IEEE J Biomed Heal Informatics 20:1088–1099

    Article  Google Scholar 

  33. Papadopoulos K, Demisse G, Ghorbel E et al (2019) Localized trajectories for 2D and 3D action recognition. Sensors (Switzerland) 19:1–22

    Article  Google Scholar 

  34. Pham HH, Khoudour L, Crouzil A, Zegers P, Velastin SA (2018) Exploiting deep residual networks for human action recognition from skeletal data. Comput Vis Image Underst 170:51–66

    Article  Google Scholar 

  35. Phamila YAV, Amutha R (2013) Low complexity energy efficient very low bit-rate image compression scheme for wireless sensor network. Inf Process Lett 113:672–676

    Article  MathSciNet  MATH  Google Scholar 

  36. Phamila AVY, Amutha R (2015) Energy-efficient low bit rate image compression in wavelet domain for wireless image sensor networks. Electron Lett 51:824–826

    Article  Google Scholar 

  37. Ponuma R, Amutha R (2019) Encryption of image data using compressive sensing and chaotic system. Multimed Tools Appl 78:11857–11881

    Article  MATH  Google Scholar 

  38. Prajapati SK, Gage WH, Brooks D, Black SE, McIlroy WE (2011) A novel approach to ambulatory monitoring: investigation into the quantity and control of everyday walking in patients with subacute stroke. Neurorehabil Neural Repair 25:6–14

    Article  Google Scholar 

  39. Rautaray SS, Agrawal A (2012) Vision based hand gesture recognition for human computer interaction: a survey. Artif Intell Rev 43:1–54

    Article  Google Scholar 

  40. Scano A, Chiavenna A, Malosio M, Molinari Tosatti L (2017. Epub ahead of print) Kinect V2 performance assessment in daily-life gestures: cohort study on healthy subjects for a reference database for automated instrumental evaluations on neurological patients. Appl Bionics Biomech 2017:2017–2016. https://doi.org/10.1155/2017/8567084

    Article  Google Scholar 

  41. Scano A, Chiavenna A, Malosio M, Molinari Tosatti L, Molteni F (2018) Kinect V2 implementation and testing of the reaching performance scale for motor evaluation of patients with neurological impairment. Med Eng Phys 56:54–58

    Article  Google Scholar 

  42. Scherer M, Unterbrunner A, Riess B, Kafka P (2016) Development of a system for supervised training at home with Kinect V2. Procedia Eng 147:466–471

    Article  Google Scholar 

  43. Simonyan K, Zisserman A (2014) Two-stream convolutional networks for action recognition in videos. In Advances in neural information processing systems:568–576

  44. Singh A, Sharma LN, Dandapat S (2016) Multi-channel ECG data compression using compressed sensing in eigenspace. Comput Biol Med 73:24–37

    Article  Google Scholar 

  45. Sun B, Zhang Z, Liu X, Hu B, Zhu T (2017) Self-esteem recognition based on gait pattern using Kinect. Gait Posture 58:428–432

    Article  Google Scholar 

  46. Tannous H, Istrate D, Benlarbi-Delai A, et al. (2016) A new multi-sensor fusion scheme to improve the accuracy of knee flexion kinematics for functional rehabilitation movements. Sensors (Switzerland); 16. Epub ahead of print. https://doi.org/10.3390/s16111914.

  47. Theodorakopoulos I, Kastaniotis D, Economou G, Fotopoulos S (2014) Pose-based human action recognition via sparse representation in dissimilarity space. J Vis Commun Image Represent 25:12–23

    Article  Google Scholar 

  48. Uswatte G, Giuliani C, Winstein C, Zeringue A, Hobbs L, Wolf SL (2006) Validity of Accelerometry for monitoring real-world arm activity in patients with subacute stroke: evidence from the extremity constraint-induced therapy evaluation trial. Arch Phys Med Rehabil 87:1340–1345

    Article  Google Scholar 

  49. Vemulapalli R, Arrate F, Chellappa R (2014) Human action recognition by representing 3D skeletons as points in a lie group. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit:588–595

  50. Vieira AW, Nascimento ER, Oliveira GL, Liu Z, Campos MFM (2014) On the improvement of human action recognition from depth map sequences using space-time occupancy patterns. Pattern Recogn Lett 36:221–227

    Article  Google Scholar 

  51. Wang P, Li W, Gao Z, Zhang J, Tang C, Ogunbona PO (2016) Action recognition from depth maps using deep convolutional neural networks. IEEE Trans Human-Machine Syst 46:498–509

    Article  Google Scholar 

  52. Yang Z, Li Y, Yang J et al (2018) Action recognition with Spatio-temporal visual attention on skeleton image sequences. IEEE Trans Circuits Syst Video Technol 1:1–10

    Google Scholar 

  53. Zeng K, Yan J, Wang Y, Sik A, Ouyang G, Li X (2016) Automatic detection of absence seizures with compressive sensing EEG. Neurocomputing 171:497–502

    Article  Google Scholar 

  54. Zhang Y, Zhang Y, Zhang Z, et al. (2018) Human activity recognition based on time series analysis using U-Net, http://arxiv.org/abs/1809.08113

  55. Zhou N, Pan S, Cheng S, Zhou Z (2016) Image compression-encryption scheme based on hyper-chaotic system and 2D compressive sensing. Opt Laser Technol 82:121–133

    Article  Google Scholar 

  56. Zhou B, Sun Y, Bau D, et al. (2018) Revisiting the importance of individual units in CNNs via ablation, http://arxiv.org/abs/1806.02891

Download references

Author information

Authors and Affiliations

  1. Department of Electronics and Communication, Sri Sivasubramaniya Nadar College of Engineering, Chennai, India

    K Ashwini & R Amutha

Authors
  1. K Ashwini
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. R Amutha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to K Ashwini.

Ethics declarations

Conflict of interest

The authors Ashwini K and Amutha R declare that they have no conflict of interest.

Ethical approval

The procedures involved in the study involving human participants were performed in accordance with the ethical standards of the institutional committee and with the 1964 Helsinki declaration and its later amendments.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ashwini, K., Amutha, R. Compressive sensing based recognition of human upper limb motions with kinect skeletal data. Multimed Tools Appl 80, 10839–10857 (2021). https://doi.org/10.1007/s11042-020-10327-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-10327-4

Keywords

Search

Navigation