User performance evaluation and real-time guidance in cloud-based physical therapy monitoring and guidance system

  • Wenchuan Wei
  • Yao Lu
  • Eric Rhoden
  • Sujit Dey


The effectiveness of traditional physical therapy may be limited by the sparsity of time a patient can spend with the physical therapist (PT) and the inherent difficulty of self-training given the paper/figure/video instructions provided to the patient with no way to monitor and ensure compliance with the instructions. In this paper, we propose a cloud-based physical therapy monitoring and guidance system. It is able to record the actions of the PT as he/she demonstrates a task to the patient in an offline session, and render the PT as an avatar. The patient can later train himself by following the PT avatar and getting real-time guidance on his/her device. Since the PT and user (patient) motion sequences may be misaligned due to human reaction and network delays, we propose a Gesture-Based Dynamic Time Warping algorithm that can segment the user motion sequence into gestures, and align and evaluate the gesture sub-sequences, all in real time. We develop an evaluation model to quantify user performance based on different criteria provided by the PT for a task, trained with offline subjective test data consisting of user performance and physical therapist scores. Moreover, we design three types of guidance which can be provided after each gesture based on user score, and conduct subjective tests to validate their effectiveness. Experiments with multiple subjects show that the proposed system can effectively train patients, give accurate evaluation scores, and provide real-time guidance which helps the patients learn the tasks and reach the satisfactory score with less time.


Dynamic time warping Gesture segmentation Motion data alignment Physical therapy Real-time guidance 



This work is funded by the National Science Foundation under grant number IIS-1522125.


  1. 1.
    Alexiadis DS et al (2011) Evaluating a dancer's performance using kinect-based skeleton tracking. In: Proc. of the 19th ACM international conference on Multimedia (MM’11). ScottsdaleGoogle Scholar
  2. 2.
    Ali Z, Muhammad G, Alhamid MF (2017) An automatic health monitoring system for patients suffering from voice complications in smart cities. IEEE Access 5:3900–3908CrossRefGoogle Scholar
  3. 3.
    Amazon Web Services. [Online]. Available:
  4. 4.
    Ananthanarayan S et al (2013) Pt Viz: towards a wearable device for visualizing knee rehabilitation exercises. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’13). ParisGoogle Scholar
  5. 5.
    Anderson F, Grossman T, Matejka J, Fitzmaurice G (2013) YouMove: enhancing movement training with an augmented reality mirror. In: Proceedings of the 26th annual ACM symposium on User interface software and technology (UIST’13). St AndrewsGoogle Scholar
  6. 6.
    Aziz K et al (2016) Smart real-time healthcare monitoring and tracking system using GSM/GPS technologies. In: Big Data and Smart City (ICBDSC’16). MuscatGoogle Scholar
  7. 7.
    Bau O, Mackay WE (2008) OctoPocus: a dynamic guide for learning gesture-based command sets. In: Proceedings of the 21st annual ACM symposium on User interface software and technology (UIST’08). MontereyGoogle Scholar
  8. 8.
    Berndt DJ, Clifford J (1994) Using Dynamic Time Warping to Find Patterns in Time Series. In: KDD workshop, vol 10, no. 16Google Scholar
  9. 9.
    Catarinucci L et al (2015) An IoT-aware architecture for smart healthcare systems. IEEE Internet of Things Journal 2(6):515–526CrossRefGoogle Scholar
  10. 10.
    Chang YJ, Chen SF, Huang JD (2011) A Kinect-based system for physical rehabilitation: a pilot study for young adults with motor disabilities. Res Dev Disabil 32(6):2566–2570CrossRefGoogle Scholar
  11. 11.
    Chang CY et al (2012) Towards pervasive physical rehabilitation using Microsoft Kinect. In: Pervasive Computing Technologies for Healthcare (PervasiveHealth’12). San DiegoGoogle Scholar
  12. 12.
    Choden P et al (2017) Volatile urine biomarkers detection in type II diabetes towards use as smart healthcare application. In: Knowledge and Smart Technology (KST’17). ChonburiGoogle Scholar
  13. 13.
    Dey S, Liu Y, Wang S, Lu Y (2013) Addressing response time of cloud-based mobile applications. In: Proceedings of the first international workshop on Mobile cloud computing & networking. ACMGoogle Scholar
  14. 14.
    Doyle J, Bailey C, Dromey B, Scanaill CN (2010) BASE-An interactive technology solution to deliver balance and strength exercises to older adults. In: 2010 4th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth’10). MunichGoogle Scholar
  15. 15.
    Freeman D, Benko H, Morris MR, Wigdor D (2009) ShadowGuides: visualizations for in-situ learning of multi-touch and whole-hand gestures. In: Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS’09). BanffGoogle Scholar
  16. 16.
    Gwet KL (2008) Intrarater reliability. Wiley encyclopedia of clinical trials. John Wiley and Sons, Hoboken, pp. 1–14Google Scholar
  17. 17.
    Jack D et al (2001) Virtual reality-enhanced stroke rehabilitation. Neural Systems and Rehabilitation Engineering, IEEE Transactions on 9(3):308–318CrossRefMathSciNetGoogle Scholar
  18. 18.
    Kahol K, Tripathi P, Panchanathan S (2004) Automated gesture segmentation from dance sequences. In: Proceedings of the Sixth IEEE International conference on Automatic Face and Gesture Recognition (FGR’04). SeoulGoogle Scholar
  19. 19.
    Kim D, Song J, Kim D (2007) Simultaneous gesture segmentation and recognition based on forward spotting accumulative HMMs. Pattern Recogn 40(11):3012–3026CrossRefzbMATHGoogle Scholar
  20. 20.
    Kinect. [Online]. Available:
  21. 21.
    Lange B et al (2011) Development and evaluation of low cost game-based balance rehabilitation tool using the Microsoft Kinect sensor. In: Engineering in Medicine and Biology Society (EMBC’11). BostonGoogle Scholar
  22. 22.
    Linktropy. [Online]. Available:
  23. 23.
    Lu Y, Liu Y, Dey S (2015) Cloud mobile 3D display gaming user experience modeling and optimization by asymmetric graphics rendering. IEEE Journal of Selected Topics in Signal Processing 9(3):517–532CrossRefGoogle Scholar
  24. 24.
    Mirelman A, Patritti BL, Bonato P, Deutsch JE (2010) Effects of virtual reality training on gait biomechanics of individuals post-stroke. Gait & posture 31(4):433–437CrossRefGoogle Scholar
  25. 25.
    Müller M (2007) Information retrieval for music and motion, vol 2. Springer, HeidelbergCrossRefGoogle Scholar
  26. 26.
    Nkosi MT, Mekuria F (2010) Cloud computing for enhanced mobile health applications. In: Cloud Computing Technology and Science (CloudCom’10). IndianapolisGoogle Scholar
  27. 27.
    Seber GA, Lee AJ (2012) Linear regression analysis, vol 936. Wiley, New YorkzbMATHGoogle Scholar
  28. 28.
    Sodhi R, Benko H, Wilson A (2012) LightGuide: projected visualizations for hand movement guidance. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’12). AustinGoogle Scholar
  29. 29.
    Tang R, Yang XD, Bateman S, Jorge J, Tang A (2015) Physio@ Home: Exploring visual guidance and feedback techniques for physiotherapy exercises. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI’15). SeoulGoogle Scholar
  30. 30.
    The Microsoft documentation for Kinect 2.0. [Online]. Available:
  31. 31.
  32. 32.
    Unity. [Online]. Available:
  33. 33.
    Wei W, Lu Y, Printz C, Dey S (2015) Motion Data Alignment and Real-Time Guidance in Cloud-Based Virtual Training System. In: Proc. of Wireless Health (WH'15). BethesdaGoogle Scholar
  34. 34.
    Yang Z et al (2016) An IoT-cloud based wearable ECG monitoring system for smart healthcare. J Med Syst 40(12):286CrossRefGoogle Scholar
  35. 35.
    Yurtman A, Barshan B (2014) Detection and evaluation of physical therapy exercises by dynamic time warping using wearable motion sensor units. In: Information Sciences and Systems (SIU’14). TrabzonGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  1. 1.Mobile Systems Design Lab, Department of Electrical and Computer EngineeringUniversity of California San DiegoLa JollaUSA
  2. 2.Department of Rehabilitation ServicesUniversity of California San DiegoLa JollaUSA

Personalised recommendations