Multimedia Tools and Applications

, Volume 76, Issue 6, pp 8227–8255 | Cite as

Web-based multimedia hand-therapy framework for measuring forward and inverse kinematic data



Recognizing rotational and angular hand movements using a non-invasive way is a challenging task. In this paper, we present a web-based multimedia hand-Therapy framework that can dynamically provide therapy services to a patient having Hemiplegia disability. Using off the shelf 3D motion control sensor called LEAP, the framework can detect, recognize and track different high-level therapeutic movements, both rotational as well as angular, originated from different joints of a Hemiplegic patient’s hand and deduce live kinematic data from these movements. The obtained kinematic data consists of a wide span of hand joint and therapy parameters that is assumed to help medical professionals in their clinical decision making. The multimedia environment uses forward kinematics to receive the live sensory data from the 3D motion tracking sensors and uses inverse kinematics to analyze the sensory data streams in real-time. A subject performs a rehabilitation therapy prescribed by the physician and using both forward and inverse kinematics, the system validates the angular and rotational motion of the joints with respect to the correct therapeutic posture and provides live feedback to the subject. The proposed method is non-invasive as the patient does not need to wear any external devices in the hand and hence can perform therapy exercises even at home. The system uses serious game interfaces to provide immersive and engaging perspective to the therapy domain. To the best of our knowledge, this study is the first of its kind to propose a non-invasive way of tracking live forward and inverse kinematic data in the context of hand therapy. We share the implementation details and our initial test results. Finally, our experiment shows that LEAP has the potential to be used for obtaining kinematic data from primitive and complex hand therapies.


Multimedia for hand-therapy Kinematic data LEAP Hand gesture tracking 


  1. 1.
    Alamri A, Kim H-N, Cha J, El Saddik A (2012) Serious games for rehabilitation of stroke patients with vibrotactile feedback. Int J Comp Sci Sport 9(2)Google Scholar
  2. 2.
    Albrecht I, Haber J, Seidel H-P (2003) Construction and animation of anatomically based human hand models. Proc SCA 98–109Google Scholar
  3. 3.
    Anderson SA, Blandine C-G (1993) Anatomy of movement. Eastland Press, Chicago. ISBN 0-939616-17-3Google Scholar
  4. 4.
    Aziz O, Atallah L, Lo B, Elhelw M, Wang L, Yang GZ, Darzi A (2007) A pervasive body sensor network for measuring postoperative recovery at home. J Surg Innov 14(2):83–90CrossRefGoogle Scholar
  5. 5.
    Baran M, Lehrer N, Siwiak D, Chen Y, Duff M, Ingalls T, Rikakis T (2011) Design of a home-based adaptive mixed reality rehabilitation system for stroke survivors. Eng Med Biol Soc EMBC7602–7605Google Scholar
  6. 6.
    Chang Y, Chen S, Huang J (2011) A Kinect-based system for physical rehabilitation: a pilot study for young adults with motor disabilities. Res Dev Disabil 32(6):2566–2570CrossRefGoogle Scholar
  7. 7.
    Chuanchu W, Kok SP, Kai KA, Cuntai G, Haihong Z, Rongsheng L, Sui GC, Geok Chua K, Beng TA, Kuah CWK (2009) A feasibility study of non-invasive motor-imagery BCI-based robotic rehabilitation for Stroke patients. 4th International IEEE/EMBS Conference on Neural Engineering. pp 271–274Google Scholar
  8. 8.
    Dolphin JA (1965) Extensor tenotomy for chronic boutonnière deformity of the finger report of two cases. J Bone Joint Surg 47(1):161–164CrossRefGoogle Scholar
  9. 9.
    Erdemir A, McLean S, Herzog W, van den Bogert AJ (2007) Model-based estimation of muscle forces exerted during movements. J Clin Biomech, Elsevier 22(2):131–154CrossRefGoogle Scholar
  10. 10.
    Erol A, Bebis G, Nicolescu M, Boyle RD, Twombly X (2007) Vision-based hand pose estimation: a review. Comp Vision Image Underst Sci Dir 108(1–2):52–57CrossRefGoogle Scholar
  11. 11.
    Gabel M, Renshaw E, Schuster A, Gilad-Bachrach R (2012) Full body gait analysis with kinect. IEEE Eng Med Biol Soc (EMBC) 1964–1967Google Scholar
  12. 12.
    Gips J, Betke M, DiMattia PA (2007) Early experiences using visual tracking for computer access by people with profound physical disabilities. Disabil Rehabil Assist Technol 2(4):189–199CrossRefGoogle Scholar
  13. 13.
    Goto A, Leng S, Sugamoto K, Cooney WP III, Kakar S, Zhao K (2014) In vivo pilot study evaluating the thumb carpometacarpal joint during circumduction. Clin Orthop Relat Res 472(4):1106–1113Google Scholar
  14. 14.
    Gustus A, Stillfried G, Visser J, Jörntell H, van der Smagt P (2012) Human hand modelling: kinematics, dynamics, applications. J Biol Cybern, Springer 106:741–755MathSciNetCrossRefMATHGoogle Scholar
  15. 15.
    Hanna SE, Bartlett DJ, Rivard LM, Russell DJ (2008) Reference curves for the gross motor function measure: percentiles for clinical description and tracking over time among children with cerebral palsy. Phys Ther 88(5):596–607CrossRefGoogle Scholar
  16. 16.
    Harley L, Robertson S, Gandy M, Harbert S, Britton D (2011) The design of an interactive stroke rehabilitation gaming system. Proc. of the 14th Intl. Conf. Human-computer interaction: users and applications, Orlando FL, pp 167–173Google Scholar
  17. 17.
    Herman IP (2007) Physics of human body. Springer, ISBN-10 3-540-29603-4Google Scholar
  18. 18.
    Holden MK (2005) Virtual environments for motor rehabilitation: review. Cyber Psychol Behav 8(3):187–211CrossRefGoogle Scholar
  19. 19.
    Hondori HM, Khademi M, Dodakian L, Cramer SC, Lopes CV (2013) A spatial augmented reality rehab system for post-stroke hand rehabilitation In proceeding of 2013 Conference on Medicine Meets Virtual Reality, NextMed/MMVR20Google Scholar
  20. 20.
    Hovet L, Ryall K, Mcdonnell R, O’Sullivan C (2012) Sleight of hand: perception of finger motion from reduced marker sets. Proc. of the ACM Symposium on Interactive 3D Graphics and Games, pp 79–86Google Scholar
  21. 21.
    Hume MC, Gellman H, McKellop H, Brumfield RH (1990) Functional range of motion of the joints of the hand. J Hand Surg 15(2):240–243CrossRefGoogle Scholar
  22. 22.
    Industrial-robotics-theory-modelling-control. In: Cubero S (ed) ARS/plV, Germany, ISBN 3-86611-285-8, p 964Google Scholar
  23. 23.
    Jenkins M, Bamberger HB, Black L, Nowinski R (1998) Thumb joint motion: what is normal? J Hand Surg 23(6):796–797CrossRefGoogle Scholar
  24. 24.
    Khoshelham K, Elberink SO (2012) Accuracy and resolution of kinect depth data for indoor mapping applications. Sensors 12:1437–1454CrossRefGoogle Scholar
  25. 25.
    Kim D, Hilliges O, Iazddi S, Butler A, Chen J, Oikonomidis I, Olivier P (2012) Digits: free 3D interactions anywhere using a wrist-worn gloveless sensor dithered color quantization. Proc ACM UIST 167–176Google Scholar
  26. 26.
    LaBelle K (2011) Evaluation of kinect joint tracking for clinical and in-home stroke rehabilitation tools. Undergraduate thesis, Notre Dame, IndianaGoogle Scholar
  27. 27.
    Lange B, Koenig S, McConnell E, Chang C, Juang R, Suma E, Bolas M, Rizzo A (2012) Interactive game-based rehabilitation using the microsoft kinect. IEEE Virt Real 171–172Google Scholar
  28. 28.
    Ma M, Bechkoum K Serious games for movement therapy after stroke systems. In Proceedings of IEEE International Conference on Man and Cybernetics. Singapore, 1872–1877Google Scholar
  29. 29.
    Mallon WJ, Brown HR, Nunley JA (1991) Digital ranges of motion: normal values in young adults. J Hand Surg 16A:882CrossRefGoogle Scholar
  30. 30.
    Mehm F, Hardy S, Göbel S, Steinmetz R (2011) Collaborative authoring of serious games for health. Proc. of the 19th ACM international conference on Multimedia. ACM, New York, pp 807–808Google Scholar
  31. 31.
    Minear WL (1956) A classification of cerebral palsy, pediatrics. Am Acad Pediatr 18(5):841–852Google Scholar
  32. 32.
    Rahman MA, Basalamah S, Toonsi AH, El Saddik A (2013) A Framework toward detecting and visualizing kinematic data for children with hemiplegia. IEEE Healthcom, Lisbon, pp 9–12Google Scholar
  33. 33.
    Rahman MA, El Saddik A (2013) Modeling e-health framework towards personalized and context-aware multimedia services. In: Furht B, Agarwal A (eds) Handbook of innovative medical technologies, Springer, ISBN 978-1-4614-8494-3Google Scholar
  34. 34.
    Rahman MA, Qamar AM, Ahmed MA, Rahman MA, Basalamah S (2013) Multimedia interactive therapy environment for children having physical disabilities. International Conference on Multimedia Retrieval, Dallas, pp 16–19Google Scholar
  35. 35.
    Saini S, Sulaiman S, Zakaria M, Shukri S (2012) A Low-cost game framework for a home-based stroke rehabilitation system. Proc. of International Conference on Computer and Information Science (ICCIS), pp 55–60Google Scholar
  36. 36.
    Skarilova B, Plevkova A (1996) Ranges of joint motion of the adult hand. Acta Chir Plast 38(67)Google Scholar
  37. 37.
    Smahel Z, Klimova A (2004) The influence of age and exercise on the mobility of hand joints 1: metacarpophalangeal joint of the threephalangeal fingers. Acta Chir Plast 46(81)Google Scholar
  38. 38.
    Smahel Z, Klimova A (2004) The influence of age and exercise on the mobility of hand joints 2: interphalangeal joints of the three-phalangeal fingers. Acta Chir Plast 46(122)Google Scholar
  39. 39.
    Stanley FJ, Blair E, Alberman E (2000) Cerebral palsies: epidemiology and causal pathways. MacKeith Press, LondonGoogle Scholar
  40. 40.
    Stone EE, Skubic M (2011) Passive in-home measurement of stride-to-stride gait variability comparing vision and Kinect sensing. Proc. of IEEE Engineering in Medicine and Biology Society (EMBC). Boston, USA, pp 6491–6494Google Scholar
  41. 41.
    Sueda S, Kaufman A, PAI DK (2008) Musculotendon simulation for hand animation. ACM Trans Graph 27(3)Google Scholar
  42. 42.
    Tsang W, Singh K, Eugene F (2005) Helping hand: an anatomically accurate inverse dynamics solution for unconstrained hand motion. Proc SCA 319–328Google Scholar
  43. 43.
    Weichert F, Bachmann D, Rudak B, Fisseler D (2013) Analysis of the accuracy and robustness of the leap motion controller. Sensors 13:6380–6393CrossRefGoogle Scholar
  44. 44.
    Winter DA (2009) Biomechanics and motor control of human movement. John Wiley & Sons, ISBN: 978-0-470-39818-0Google Scholar
  45. 45.
    Zhao W, Chai J, Xu Y (2012) Combining marker-based mocap and RGB-D camera for acquiring high-fidelity hand motion data. ACM SIGGRAPH 33–42Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.Advanced Media Laboratory, Department of Computer Science, College of Computer and Information SystemsUmm Al-Qura UniversityMakkahKingdom of Saudi Arabia

Personalised recommendations