Multimedia Tools and Applications

, Volume 78, Issue 22, pp 32055–32085 | Cite as

Continues online exercise monitoring and assessment system with visual guidance feedback for stroke rehabilitation

  • Fatemeh Mortazavi
  • Ali Nadian-GhomshehEmail author


Exercise therapy is a conventional intervention for stroke rehabilitation. Performance monitoring and feedback have shown to further improve the outcome of exercise therapy. This paper proposes a vision based system for monitoring exercise therapy which consists of 3 components: online exercise recognition, exercise performance analysis, and automatic visual feedback generation. The Microsoft Kinect was used for data acquisition. The exercise recognition component utilizes Kinect joints to continuously recognize and track the exercises. Upon completion of each exercise, joint flexibility and compensatory trunk motions are extracted for performance analysis. The visual feedback is a virtual skeleton augmented on top of the Kinect skeleton which displays the correct exercise path during execution. The Kinect skeleton and exercise definitions were applied to a motion hierarchy and animated using forward kinematics. Two additional experiments were also conducted to find accurate methods for calculating joint flexibility based on ROM measurement and trunk representation. Several datasets were created for system design and evaluation: 336 exercise sequences for exercise recognition, 25 records for ROM measurement, and 63 records for finding a suitable trunk representation method and compensatory motion detection. System evaluations showed that each component of the system is capable of producing outputs with significant accuracy.


Action recognition Exercise therapy Range of motion Motion hierarchy Forward kinematics Compensatory motion Rehabilitation 


Supplementary material

11042_2019_8020_MOESM1_ESM.avi (7.8 mb)
ESM 1 (AVI 7936 kb)
11042_2019_8020_MOESM2_ESM.avi (5.6 mb)
ESM 2 (AVI 5727 kb)
11042_2019_8020_MOESM3_ESM.avi (5.2 mb)
ESM 3 (AVI 5349 kb)
11042_2019_8020_MOESM4_ESM.avi (3.5 mb)
ESM 4 (AVI 3548 kb)


  1. 1.
    Abdollahi F, Case Lazarro ED, Listenberger M, Kenyon RV, Kovic M, Bogey RA, Hedeker D, Jovanovic BD, Patton JL (2014) Error augmentation enhancing arm recovery in individuals with chronic stroke: a randomized crossover design. Neurorehabil Neural Repair 28:120–128CrossRefGoogle Scholar
  2. 2.
    Aggarwal JK, Xia L (2014) Human activity recognition from 3d data: A review. Pattern Recogn Lett 48:70–80CrossRefGoogle Scholar
  3. 3.
    Ayoade M, Baillie L (2014) A novel knee rehabilitation system for the home. In: Proceedings of the 32nd annual ACM conference on Human factors in computing systems, 2014. ACM, pp 2521-2530Google Scholar
  4. 4.
    Barnachon M, Bouakaz S, Boufama B, Guillou E (2014) Ongoing human action recognition with motion capture. Pattern Recogn 47(1):238–247CrossRefGoogle Scholar
  5. 5.
    Bloom V, Argyriou V, Makris D (2017) Linear latent low dimensional space for online early action recognition and prediction. Pattern Recogn 72:532–547CrossRefGoogle Scholar
  6. 6.
    Bonnechere B, Jansen B, Salvia P, Bouzahouene H, Omelina L, Moiseev F, Sholukha V, Cornelis J, Rooze M, Jan SVS (2014) Validity and reliability of the Kinect within functional assessment activities: comparison with standard stereophotogrammetry. Gait Posture 39:593–598CrossRefGoogle Scholar
  7. 7.
    Chandra H, Oakley I, Silva H (2012) Designing to support prescribed home exercises: understanding the needs of physiotherapy patients. In: Proceedings of the 7th Nordic Conference on Human-Computer Interaction: Making Sense Through Design. ACM, pp 607-616Google Scholar
  8. 8.
    Chang Y-J, Chen S-F, Huang J-D (2011) A Kinect-based system for physical rehabilitation: A pilot study for young adults with motor disabilities. Res Dev Disabil 32:2566–2570CrossRefGoogle Scholar
  9. 9.
    Chang Y-J, Han W-Y, Tsai Y-C (2013) A Kinect-based upper limb rehabilitation system to assist people with cerebral palsy. Res Dev Disabil 34:3654–3659CrossRefGoogle Scholar
  10. 10.
    Chang X, Yu Y-L, Yang Y, Xing EP (2017) Semantic pooling for complex event analysis in untrimmed videos. IEEE Trans Pattern Anal Mach Intell 39(8):1617–1632CrossRefGoogle Scholar
  11. 11.
    Conner C, Poor GM (2016) Correcting exercise form using body tracking. In: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, 2016. ACM, pp 3028-3034Google Scholar
  12. 12.
    Conradsson D, Nero H, Löfgren N, Hagströmer M, Franzén E (2017) Monitoring training activity during gait-related balance exercise in individuals with Parkinson’s disease: a proof-of-concept-study. BMC Neurol 17:19CrossRefGoogle Scholar
  13. 13.
    Crasto JA, Sayari AJ, Gray RR, Askari M (2015) Comparative analysis of photograph-based clinical goniometry to standard techniques. Hand 10:248–253CrossRefGoogle Scholar
  14. 14.
    Da Gama A, Chaves T, Figueiredo L, Teichrieb V (2012) Guidance and movement correction based on therapeutics movements for motor rehabilitation support systems. In: Virtual and Augmented Reality (SVR), 2012 14th Symposium on, 2012. IEEE, pp 191-200Google Scholar
  15. 15.
    Deters JK, Rybarczyk Y (2018) Hidden Markov Model approach for the assessment of tele-rehabilitation exercises. International Journal of Artificial Intelligence 16(1):1–19Google Scholar
  16. 16.
    English C, Bernhardt J, Crotty M, Esterman A, Segal L, Hillier S (2015) Circuit class therapy or seven-day week therapy for increasing rehabilitation intensity of therapy after stroke (CIRCIT): a randomized controlled trial. Int J Stroke 10:594–602CrossRefGoogle Scholar
  17. 17.
    Flanders M, Kavanagh RC (2015) Build-A-Robot: Using virtual reality to visualize the Denavit–Hartenberg parameters. Comput Appl Eng Educ 23:846–853CrossRefGoogle Scholar
  18. 18.
    Gajdosik RL, Bohannon RW (1987) Clinical measurement of range of motion: review of goniometry emphasizing reliability and validity. Phys Ther 67:1867–1872CrossRefGoogle Scholar
  19. 19.
    Gal N, Andrei D, Nemeş DI, Nădăşan E, Stoicu-Tivadar V (2015) A Kinect based intelligent e-rehabilitation system in physical therapy. Studies in Health Technology and Informatics 210:489–493Google Scholar
  20. 20.
    Gauthier LV, Kane C, Borstad A, Strahl N, Uswatte G, Taub E, Morris D, Hall A, Arakelian M, Mark V (2017) Video Game Rehabilitation for Outpatient Stroke (VIGoROUS): protocol for a multi-center comparative effectiveness trial of in-home gamified constraint-induced movement therapy for rehabilitation of chronic upper extremity hemiparesis. BMC Neurol 17:109CrossRefGoogle Scholar
  21. 21.
    Han F, Reily B, Hoff W, Zhang H (2017) Space-time representation of people based on 3D skeletal data: A review. Comput Vis Image Underst 158:85–105CrossRefGoogle Scholar
  22. 22.
    Hiraoka K (2001) Rehabilitation effort to improve upper extremity function in post-stroke patients: a meta-analysis. J Phys Ther Sci 13:5–9CrossRefGoogle Scholar
  23. 23.
    Hsieh C-L, Sheu C-F, Hsueh I-P, Wang C-H (2002) Trunk control as an early predictor of comprehensive activities of daily living function in stroke patients. Stroke 33:2626–2630CrossRefGoogle Scholar
  24. 24.
    Kim E, Kim K (2015) Effect of purposeful action observation on upper extremity function in stroke patients. J Phys Ther Sci 27:2867–2869CrossRefGoogle Scholar
  25. 25.
    Kitsunezaki N, Adachi E, Masuda T, Mizusawa J-I (2013) KINECT applications for the physical rehabilitation. In: Medical Measurements and Applications Proceedings (MeMeA), 2013 IEEE International Symposium on, 2013. IEEE, pp 294–299Google Scholar
  26. 26.
    Lam MY, Tatla SK, Lohse KR, Shirzad N, Hoens AM, Miller KJ, Holsti L, Virji-Babul N, Van der Loos HM (2015) Perceptions of technology and its use for therapeutic application for individuals with hemiparesis: findings from adult and pediatric focus groups. JMIR Rehabilitation and Assistive Technologies 2Google Scholar
  27. 27.
    Lam K-Y, Tsang NW-H, Han S, Zhang W, Ng JK-Y, Nath A (2017) Activity tracking and monitoring of patients with Alzheimer’s disease. Multimed Tools Appl 76:489–521CrossRefGoogle Scholar
  28. 28.
    Lam AW, Varona-Marin D, Li Y, Fergenbaum M, Kulić D (2016) Automated rehabilitation system: Movement measurement and feedback for patients and physiotherapists in the rehabilitation clinic. Human–Computer Interaction 31:294–334CrossRefGoogle Scholar
  29. 29.
    Lee K-H (2015) The role of compensatory movements patterns in spontaneous recovery after stroke. J Phys Ther Sci 27:2671–2673CrossRefGoogle Scholar
  30. 30.
    Levin MF, Kleim JA, Wolf SL (2009) What do motor “recovery” and “compensation” mean in patients following stroke? Neurorehabil Neural Repair 23(4):313–319CrossRefGoogle Scholar
  31. 31.
    Li Z, Nie F, Chang X, Yang Y (2017) Beyond trace ratio: weighted harmonic mean of trace ratios for multiclass discriminant analysis. IEEE Trans Knowl Data Eng 29(10):2100–2110CrossRefGoogle Scholar
  32. 32.
    Manghisi VM, Uva AE, Fiorentino M, Bevilacqua V, Trotta GF, Monno G (2017) Real time RULA assessment using Kinect v2 sensor. Appl ErgonGoogle Scholar
  33. 33.
    Mastos M, Miller K, Eliasson A-C, Imms C (2007) Goal-directed training: linking theories of treatment to clinical practice for improved functional activities in daily life. Clin Rehabil 21:47–55CrossRefGoogle Scholar
  34. 34.
    Mousavi Hondori H, Khademi M (2014) A review on technical and clinical impact of microsoft kinect on physical therapy and rehabilitation. Journal of Medical Engineering 2014Google Scholar
  35. 35.
    Mündermann L, Corazza S, Andriacchi TP (2006) The evolution of methods for the capture of human movement leading to markerless motion capture for biomechanical applications. Journal of NeuroEngineering and Rehabilitation 3:6CrossRefGoogle Scholar
  36. 36.
    Murray RM (2017) A mathematical introduction to robotic manipulation. CRC Press, Inc., Boca RatonGoogle Scholar
  37. 37.
    Nowozin S, Shotton J (2012) Action points: A representation for low-latency online human action recognition. Microsoft Research Cambridge, Tech. Rep. MSR-TR-2012-68.Google Scholar
  38. 38.
    Piron L, Turolla A, Agostini M, Zucconi CS, Ventura L, Tonin P, Dam M (2010) Motor learning principles for rehabilitation: a pilot randomized controlled study in poststroke patients. Neurorehabil Neural Repair 24:501–508CrossRefGoogle Scholar
  39. 39.
    Presti LL, La Cascia M (2016) 3D skeleton-based human action classification: A survey. Pattern Recogn 53:130–147CrossRefGoogle Scholar
  40. 40.
    Qamar A, Rahman MA, Basalamah S (2014) Adding inverse kinematics for providing live feedback in a serious game-based rehabilitation system. In: Intelligent Systems, Modelling and Simulation (ISMS), 2014 5th International Conference on, 2014. IEEE, pp 215-220Google Scholar
  41. 41.
    Radomski MV, Latham CAT (2008) Occupational therapy for physical dysfunction. Wolters Kluwer Health/Lippincott Williams & Wilkins, PhiladelphiaGoogle Scholar
  42. 42.
    Rahman MA (2015) Multimedia environment toward analyzing and visualizing live kinematic data for children with Hemiplegia. Multimed Tools Appl 74:5463–5487CrossRefGoogle Scholar
  43. 43.
    Ranganathan R, Wang R, Gebara R, Biswas S (2017) Detecting Compensatory Trunk Movements in Stroke Survivors using a Wearable System. In: Proceedings of the 2017 Workshop on Wearable Systems and Applications, 2017. ACM, pp 29–32Google Scholar
  44. 44.
    Raptis M, Kirovski D, Hoppe H (2011) Real-time classification of dance gestures from skeleton animation. In: Proceedings of the 2011 ACM SIGGRAPH/Eurographics symposium on computer animation. ACM, pp 147-156Google Scholar
  45. 45.
    Reither LR, Foreman MH, Migotsky N, Haddix C, Engsberg JR (2017) Upper extremity movement reliability and validity of the Kinect version 2. Disability and Rehabilitation: Assistive Technology 1–9Google Scholar
  46. 46.
    Roby-Brami A, Feydy A, Combeaud M, Biryukova E, Bussel B, Levin M (2003) Motor compensation and recovery for reaching in stroke patients. Acta Neurol Scand 107:369–381CrossRefGoogle Scholar
  47. 47.
    Rocha C, Tonetto C, Dias A (2011) A comparison between the Denavit–Hartenberg and the screw-based methods used in kinematic modeling of robot manipulators. Robot Comput Integr Manuf 27:723–728CrossRefGoogle Scholar
  48. 48.
    Samad R, Bakar MZA, Pebrianti D, Mustafa M, Abdullah NRH (2017) Elbow flexion and extension rehabilitation exercise system using marker-less kinect-based method. International Journal of Electrical and Computer Engineering (IJECE) 7(3):1602–1610CrossRefGoogle Scholar
  49. 49.
    Schmidt RA, Young DE (1991) Methodology for motor learning: a paradigm for kinematic feedback. J Mot Behav 23:13–24CrossRefGoogle Scholar
  50. 50.
    Seidenari L, Varano V, Berretti S, Bimbo A (2013) Pala P Recognizing actions from depth cameras as weakly aligned multi-part bag-of-poses. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. pp 479–485Google Scholar
  51. 51.
    Shotton J, Sharp T, Kipman A, Fitzgibbon A, Finocchio M, Blake A, Cook M, Moore R (2013) Real-time human pose recognition in parts from single depth images. Commun ACM 56:116–124CrossRefGoogle Scholar
  52. 52.
    Snell RS (2007) Clinical anatomy by systems. Wolters Kluwer Health/Lippincott Williams & Wilkins, PhiladelphiaGoogle Scholar
  53. 53.
    Stief F, Böhm H, Ebert C, Döderlein L, Meurer A (2014) Effect of compensatory trunk movements on knee and hip joint loading during gait in children with different orthopedic pathologies. Gait Posture 39:859–864CrossRefGoogle Scholar
  54. 54.
    Sun C, Zhang T, Xu C (2015) Latent support vector machine modeling for sign language recognition with Kinect. ACM Transactions on Intelligent Systems and Technology (TIST) 6:20Google Scholar
  55. 55.
    Tatla SK, Shirzad N, Lohse KR, Virji-Babul N, Hoens AM, Holsti L, Li LC, Miller KJ, Lam MY, Van der Loos HM (2015) Therapists’ perceptions of social media and video game technologies in upper limb rehabilitation. JMIR Serious GamesGoogle Scholar
  56. 56.
    Timmermans AA, Spooren AI, Kingma H, Seelen HA (2010) Influence of task-oriented training content on skilled arm-hand performance in stroke: a systematic review. Neurorehabil Neural Repair 24:858–870CrossRefGoogle Scholar
  57. 57.
    Valdés BA, Schneider AN, Van der Loos HM (2017) Reducing Trunk Compensation in Stroke Survivors: A Randomized Crossover Trial Comparing Visual and Force Feedback Modalities. Arch Phys Med Rehabil 98:1932–1940CrossRefGoogle Scholar
  58. 58.
    Van Vliet PM, Wulf G (2006) Extrinsic feedback for motor learning after stroke: what is the evidence? Disabil Rehabil 28:831–840CrossRefGoogle Scholar
  59. 59.
    Veerbeek JM, van Wegen E, van Peppen R, van der Wees PJ, Hendriks E, Rietberg M, Kwakkel G (2014) What is the evidence for physical therapy poststroke? A systematic review and meta-analysis. PLoS One 9:e87987CrossRefGoogle Scholar
  60. 60.
    Wang Q, Kurillo G, Ofli F, Bajcsy R (2015) Evaluation of pose tracking accuracy in the first and second generations of microsoft kinect. In: Healthcare Informatics (ICHI), 2015 International Conference on, 2015. IEEE, pp 380-389Google Scholar
  61. 61.
    Wolf SL, Sahu K, Bay RC, Buchanan S, Reiss A, Linder S, Rosenfeldt A, Alberts J (2015) The HAAPI (Home Arm Assistance Progression Initiative) trial: a novel robotics delivery approach in stroke rehabilitation. Neurorehabil Neural Repair 29:958–968CrossRefGoogle Scholar
  62. 62.
    Zankel H (1951) Photogoniometry; a new method of measurement of range of motion of joints. Arch Phys Med Rehabil 32:227Google Scholar
  63. 63.
    Zeng Z, Li Z, Cheng D, Zhang H, Zhan K, Yang Y (2018) Two-stream multirate recurrent neural network for video-based pedestrian reidentification. IEEE Transactions on Industrial Informatics 14(7):3179–3186CrossRefGoogle Scholar
  64. 64.
    Zennaro S, Munaro M, Milani S, Zanuttigh P, Bernardi A, Ghidoni S, Menegatti E (2015) Performance evaluation of the 1st and 2nd generation Kinect for multimedia applications. In: Multimedia and Expo (ICME), 2015 IEEE International Conference on, 2015. IEEE, pp 1-6Google Scholar
  65. 65.
    Zhao W, Espy DD, Reinthal MA, Feng H (2014) A feasibility study of using a single kinect sensor for rehabilitation exercises monitoring: A rule based approach. In: Computational Intelligence in Healthcare and e-health (CICARE), 2014 IEEE Symposium on, 2014. IEEE, pp 1-8Google Scholar
  66. 66.
    Zulkarnain RF, Kim G-Y, Adikrishna A, Hong HP, Kim YJ, Jeon I-H (2017) Digital data acquisition of shoulder range of motion and arm motion smoothness using Kinect v2. J Shoulder Elb Surg 26:895–901CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Cyber Space Research InstituteShahid Beheshti UniversityTehranIran

Personalised recommendations