Skip to main content
SpringerLink
Log in

HDL-PSR: Modelling Spatio-Temporal Features Using Hybrid Deep Learning Approach for Post-Stroke Rehabilitation

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Physiotherapy exercises like extension, flexion, and rotation are an absolute necessity for patients of post stroke rehabilitation (PSR). A physiotherapist uses many techniques to restore movements needs in daily life including nerve re-education, task training, muscle strengthening and uses various assistive techniques. But, a physiotherapist guiding the physiotherapy exercises to a patient is a time-consuming, tedious and costly affair. In the paper, a novel automated system is designed for detecting and recognizing upper limb exercises using an RGB-Depth camera that could guide the patients to perform real-time physiotherapy exercises without human intervention. Hybrid deep learning (HDL) approaches are exploited for the highly accurate and robust system for recognizing physiotherapy exercises of the upper limb for PSR. As a baseline, a deep convolutional neural network (CNN) is designed that automatically extracts features from the pre-processed data and classifies the performed physiotherapy exercise. As the exercise is being performed, to extract and utilize temporal dependencies, architectures of recurrent neural network (RNN) are used. In the CNN-LSTM model, CNN derives useful features that are provided to LSTM thus increasing the accuracy of recognized exercises. To train faster, another hybrid deep learning model, CNN-GRU is implemented where a novel focal loss criterion is used to overcome the drawbacks of standard cross-entropy loss. Experimental evaluation is done using RGB-D data obtained from Microsoft Kinect v2 sensors. Dataset comprising of 10 different physiotherapy exercises were created. Experimental results have shown significant activity recognition accuracy with 98% and 99% for CNN and CNN-LSTM model respectively. CNN-GRU model is the best suitable architecture with 100% accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Patil P, Kumar KS, Gaud N, Semwal VB (2019) Clinical human gait classification: extreme learning machine approach, In: 2019 1st international conference on advances in science, engineering and robotics technology (ICASERT). IEEE, pp 1–6

  2. Semwal VB, Katiyar SA, Chakraborty R, Nandi GC (2015) Biologically-inspired push recovery capable bipedal locomotion modeling through hybrid automata. Robot Auton Syst 70:181–190

    Article  Google Scholar 

  3. Gupta A, Semwal VB (2020) Multiple task human gait analysis and identification: ensemble learning approach. In: Emotion and information processing. Springer, pp 185–197

  4. Dua N, Singh SN, Semwal VB (2021) Multi-input cnn-gru based human activity recognition using wearable sensors. Computing, pp 1–18

  5. Jain R, Semwal VB, Kaushik P (2021) Deep ensemble learning approach for lower extremity activities recognition using wearable sensors. Exp Syst, p e12743

  6. Semwal VB, Gaud N, Lalwani P, Bijalwan V, Alok AK (2021) Pattern identification of different human joints for different human walking styles using inertial measurement unit (imu) sensor. Artif Intell Rev, pp 1–21

  7. Doman CA, Waddell KJ, Bailey RR, Moore JL, Lang CE (2016) Changes in upper-extremity functional capacity and daily performance during outpatient occupational therapy for people with stroke. Am J Occup Ther 70(3):1–11

    Article  Google Scholar 

  8. Crow JL, Harmeling-Van Der Wel BC (2008) Hierarchical properties of the motor function sections of the fugl-meyer assessment scale for people after stroke: a retrospective study. Phys Ther 88(12):1554–1567

    Article  Google Scholar 

  9. Bijalwan V, Semwal VB, Mandal T (2021) Fusion of multi-sensor based biomechanical gait analysis using vision and wearable sensor. IEEE Sens J

  10. İnce F, Ö, Ince IF, Yıldırım ME, Park JS, Song JK, Yoon BW, (2020) Human activity recognition with analysis of angles between skeletal joints using a RGB-depth sensor. ETRI J 42(1):78–89

  11. Jardim D, Nunes L, Dias M (2017) Human activity recognition from automatically labeled data in RGB-D videos. In: 2016 8th computer science and electronic engineering conference, CEEC 2016 - conference proceedings, pp 89–94

  12. Procházka A, Vyšata O, Vališ M, Ťupa O, Schätz M, Mařík V (2015) Use of the image and depth sensors of the microsoft kinect for the detection of gait disorders. Neural Comput Appl 26(7):1621–1629

    Article  Google Scholar 

  13. Singh G, Singh RK, Saha R, Agarwal N (2020) Iwt based iris recognition for image authentication. Procedia Comput Sci 171:1868–1876

    Article  Google Scholar 

  14. Ye M, Zhang Q, Wang L, Zhu J (2013) A survey on human motion analysis, time-of-flight and depth imaging. Sensors Algorithm Appl 8200:149–187

    Google Scholar 

  15. Shotton J, Sharp T, Fitzgibbon A, Blake A, Cook M, Kipman A, Finocchio M, Moore R (2013) Real-Time human pose recognition in parts from single depth images. Commun ACM 56(1):116–124

    Article  Google Scholar 

  16. Zhao W, Lun R, Gordon C, Fofana ABM, Espy DD, Reinthal MA, Ekelman B, Goodman GD, Niederriter JE, Luo X (2017) A human-centered activity tracking system: toward a healthier workplace. IEEE Trans Hum Mach Syst 47(3):343–355

    Article  Google Scholar 

  17. Semwal VB, Chakraborty P, Nandi GC (2015) Less computationally intensive fuzzy logic (type-1)-based controller for humanoid push recovery. Robot Auton Syst 63:122–135

    Article  Google Scholar 

  18. Singh G, Chowdhary M, Kumar A, Bahl R (2020) A personalized classifier for human motion activities with semi-supervised learning. IEEE Trans Consum Electron 66(4):346–355

    Article  Google Scholar 

  19. Singh G, Rawat T (2013) Color image enhancement by linear transformations solving out of gamut problem. Int J Comput Appl 67(14):28–32

    Google Scholar 

  20. Agarwal N, Sondhi A, Chopra K, Singh G, Transfer learning: survey and classification. In: Smart innovations in communication and computational sciences. Springer, 2021, pp 145–155

  21. Zhang L, Sheng Z, Li Y, Sun Q, Zhao Y, Feng D (2019) Image object detection and semantic segmentation based on convolutional neural network. Neural Comput Appl, pp 1–10

  22. Ramakrishnan J, Mavaluru D, Sakthivel RS, Alqahtani AS, Mubarakali A, Retnadhas M (2020) Brain–computer interface for amyotrophic lateral sclerosis patients using deep learning network. Neural Comput Appl, pp 1–15

  23. Bijalwan V, Semwal VB, Singh G, Crespo RG (2021) Heterogeneous computing model for post-injury walking pattern restoration and postural stability rehabilitation exercise recognition. Exp Syst p e12706

  24. Pham Huy-Hieu, Khoudour L, Crouzil A, Zegers P, Velastin SA (2017) Learning and recognizing human action from skeleton movement with deep residual neural networks, pp 25 (6 .)—-25 (6 .)

  25. Komang MGA, Surya MN, Ratna AN (2019) Human activity recognition using skeleton data and support vector machine. J Phys Conf Ser 1192(1)

  26. Semwal VB, Singha J, Sharma PK, Chauhan A, Behera B (2017) An optimized feature selection technique based on incremental feature analysis for bio-metric gait data classification. Multimedia Tools Appl 76(22):24457–24475

    Article  Google Scholar 

  27. Nirjon S, Greenwood C, Torres C, Zhou S, Stankovic JA, Yoon HJ, Ra HK, Basaran C, Park T, Son SH (2014) Kintense: A robust, accurate, real-time and evolving system for detecting aggressive actions from streaming 3D skeleton data. In: 2014 IEEE international conference on pervasive computing and communications, PerCom 2014, pp 2–10

  28. Chang Y-J, Chen S-F, Huang J-D (2011) A kinect-based system for physical rehabilitation: A pilot study for young adults with motor disabilities. Res Dev Disabil 32(6):2566–2570

    Article  Google Scholar 

  29. Zhao W, Reinthal MA, Espy DD, Luo X (2017) Rule-based human motion tracking for rehabilitation exercises: realtime assessment, feedback, and guidance. IEEE Access, vol 5, pp 21382–21394

  30. Semwal VB (2017) Data driven computational model for bipedal walking and push recovery, arXiv preprintarXiv:1710.06548

  31. Raj M, Semwal VB, Nandi GC (2018) Bidirectional association of joint angle trajectories for humanoid locomotion: the restricted boltzmann machine approach. Neural Comput Appl 30(6):1747–1755

    Article  Google Scholar 

  32. Caon M, Yue Y, Tscherrig J, Mugellini E, Abou Khaled O (2011) Context-aware 3D gesture interaction based on multiple kinects, AMBIENT 2011. In: The first international conference on ambient computing, applications, services and technologies, pp 7–12

  33. Singh P, Singh RK, Singh G (2018) An efficient iris recognition system using integer wavelet transform, In: 2018 2nd international conference on trends in electronics and informatics (ICOEI). IEEE, pp 1029–1034

  34. Singh RK, Saha R, Pal PK, Singh G (2018) Novel feature extraction algorithm using dwt and temporal statistical techniques for word dependent speaker’s recognition, In: 2018 fourth international conference on research in computational intelligence and communication networks (ICRCICN). IEEE, pp 130–134

  35. Semwal VB, Mondal K, Nandi GC (2017) Robust and accurate feature selection for humanoid push recovery and classification: deep learning approach. Neural Comput Appl 28(3):565–574

    Article  Google Scholar 

  36. Kanagaraj N, Hicks D, Goyal A, Tiwari S, Singh G (2021) Deep learning using computer vision in self driving cars for lane and traffic sign detection. Int J Syst Assurance Eng Manag, pp 1–15

  37. Singh G, Chowdhary M, Kumar A, Bahl R (2019) A probabilistic framework for base level context awareness of a mobile or wearable device user. In: 2019 IEEE 8th global conference on consumer electronics (GCCE). IEEE, pp 217–218

  38. Chhillar S, Singh G, Singh A, Saini VK (2019) Quantitative analysis of pulmonary emphysema by congregating statistical features, In: 2019 3rd international conference on recent developments in control, automation & power engineering (RDCAPE). IEEE, pp 329–333

  39. Zhao ZQ, Zheng P, Xu ST, Wu X (2019) Object detection with deep learning: a review. IEEE Trans Neural Netw Learn Syst 30(11):3212–3232

    Article  Google Scholar 

  40. Gupta V, Semwal VB (2021) Wearable sensor based pattern mining for human activity recognition : deep learning approach. Ind Robot, 48(1)

  41. Shao L, Han J, Xu D, Shotton J (2013) Computer vision for RGB-D sensors: kinect and its applications. IEEE Tran Cybernet 43(5):1314–1317

    Article  Google Scholar 

  42. Cao W, Zhong J, Cao G, He Z (2019) Physiological function assessment based on kinect V2. In: IEEE Access, vol 7, pp 105638–105651

  43. Collings DG, Scullion H, Vaiman V (2015) Talent management: progress and prospects. Hum Resour Manag Rev 25(3):233–235

    Google Scholar 

  44. Jalal A, Kim Y, Kamal S, Farooq A, Kim D (2015) Human daily activity recognition with joints plus body features representation using Kinect sensor. In: 2015 4th international conference on informatics, electronics and vision, ICIEV 2015,

  45. Gaglio S, Re GL, Morana M (2015) Human activity recognition process using 3-D posture data. IEEE Trans Hum Mach Syst 45(5):586–597

    Article  Google Scholar 

  46. Semwal VB, Mazumdar A, Jha A, Gaud N, Bijalwan N (2019) Speed, cloth and pose invariant gait recognition-based person identification. In: Machine learning: theoretical foundations and practical applications, p 39

  47. Gill T, Keller JM, Anderson DT, Luke RH (2011) A system for change detection and human recognition in voxel space using the Microsoft Kinect sensor. In: Proceedings - applied imagery pattern recognition workshop

  48. Su B, Wu H, Sheng M, Shen C (2019) Accurate hierarchical human actions recognition from kinect skeleton data. IEEE Access, vol7, pp 52532–52541

  49. Semwal VB, Nandi GC (2015) Toward developing a computational model for bipedal push recovery-a brief. IEEE Sens J 15(4):2021–2022

    Article  Google Scholar 

  50. Eldesokey A, Felsberg M, Khan FS (2019) Confidence propagation through cnns for guided sparse depth regression. IEEE Trans Pattern Anal Mach Intell 42(10):2423–2436

    Article  Google Scholar 

  51. He K, Sun J, Tang X (2012) Guided image filtering. IEEE Trans Pattern Anal Mach Intell 35(6):1397–1409

    Article  Google Scholar 

  52. Saha R, Singh RK, Kumar R, Singh G, Goel T, Pal PK (2019) Classification of human heart signals by novel feature extraction techniques for rescue application. In: 2019 fifth international conference on image information processing (ICIIP). IEEE, pp 156–160

  53. Pandey S, Sharma R, Singh G (2020) Implementation of 5-block convolutional neural network (cnn) for saliency improvement on flying object detection in videos. In: 2020 3rd international conference on emerging technologies in computer engineering: machine learning and internet of things (ICETCE). IEEE, pp 1–6

  54. Ahmad Z, Khan NM (2019) Multidomain multimodal fusion for human action recognition using inertial sensors. In: 2019 IEEE fifth international conference on multimedia big data (BigMM). IEEE, pp 429–434

  55. Ahmad Z, Khan NM (2020) Multidomain multimodal fusion for human action recognition using inertial sensors, CoRR, vol. abs/2008.09748. [Online]. arXiv:abs/2008.09748

Download references

Author information

Authors and Affiliations

  1. ICFAI University, Dehradun, India

    Vishwanath Bijalwan & Tapan Kumar Mandal

  2. MANIT, Bhopal, India

    Vijay Bhaskar Semwal

  3. National Institute of Technology Uttarakhand, Uttarakhand, India

    Ghanapriya Singh

Authors
  1. Vishwanath Bijalwan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vijay Bhaskar Semwal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ghanapriya Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tapan Kumar Mandal
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ghanapriya Singh.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The work is supported by grant HEFA-CSR, Government of India to Dr. Vijay Bhaskar Semwal with No: SAN/CSR/08/2021-23.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bijalwan, V., Semwal, V.B., Singh, G. et al. HDL-PSR: Modelling Spatio-Temporal Features Using Hybrid Deep Learning Approach for Post-Stroke Rehabilitation. Neural Process Lett 55, 279–298 (2023). https://doi.org/10.1007/s11063-022-10744-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-022-10744-6

Keywords

Search

Navigation