Serious Games Based on Kinect and Leap Motion Controller for Upper Limbs Physical Rehabilitation

  • Gabriela Postolache
  • Francisco Carry
  • Filipe Lourenço
  • Diogo Ferreira
  • Raul Oliveira
  • Pedro Silva Girão
  • Octavian PostolacheEmail author
Part of the Smart Sensors, Measurement and Instrumentation book series (SSMI, volume 29)


The design, implementation and tests of a system for assessment and monitoring movements, which includes the sensors from Kinect and Leap Motion Controller devices, are discussed in the present chapter. The advantages and some drawbacks of using the two devices for creating virtual environments for motor rehabilitation in which interaction of the user with virtual reality is made through natural user interfaces are described. The IoT architecture for rehabilitation environment, several serious games that our team have developed and the usability evaluation of the system are presented. Our insights, based on our research work during serious games development as well as on literature analysis, mainly focusing these on practicality of the developed serious games and on their acceptability for motor rehabilitation, are also included in the chapter. In the development of a system that includes IoT technology for movements tracking and progress evaluation during motor rehabilitation, the importance of user centered design is underscored.


Kinect Leap Motion Virtual reality Usability 



The work was supported by Fundação para a Ciência e a Tecnologia, project PTDC/DTT-DES/6776/2014 and Instituto de Telecomunicações.


  1. 1.
  2. 2.
    Pew Research Center—Number of Gamers. Statica (2018),
  3. 3.
  4. 4.
  5. 5.
  6. 6.
    C.C. Abt, Serious Games (University Press of America, New York, 1970)Google Scholar
  7. 7.
    B.P. Bergeron. Developing Serious Games, 1st edn. (Hingham Charles River Media, 2006)Google Scholar
  8. 8.
    D. Michael, S. Chen, Serious Games that Educate, Train and Inform (Thomson, Boston, MA, 2006)Google Scholar
  9. 9.
    D. Thompson, T. Baranowski, R. Buday. Serious video games for health. How behavioral science guided the development of a serious video game. Simul Gaming 41(4), 587–606 (2010)Google Scholar
  10. 10.
    D.M. Harris, T. Rantalainen, M. Muthalib, L. Johnson, W.-P. Teo, Exergaming as a viable therapeutic tool to improve static and dynamic balance among older adults and people with idiopathic Parkinson’s disease: a systematic review and meta-analysis. Front Aging Neurosci. 7(167), 1–12 (2015)Google Scholar
  11. 11.
    N. Skjæret, A. Nawaz, T. Morat, D. Schoene, J.L. Helbostad, B. Vereijken, Exercise and rehabilitation delivered through exergames in older adults: an integrative review of technologies, safety and efficacy. Int. J. Med. Inform. 85(1), 1–16 (2016)CrossRefGoogle Scholar
  12. 12.
    L. Donath, R. Rössler, O. Faude, Effects of virtual reality training (exergaming) compared to alternative exercise training and passive control on standing balance and functional mobility in healthy community-dwelling seniors: a meta-analytical review. Sports Med. 46(9), 1293–1309 (2016)CrossRefGoogle Scholar
  13. 13.
    S.G.R. Neri, J.R. Cardoso, L. Cruz, R.M. Lima, R.J. de Oliveira, M.D. Iversen, R.L. Carregaro, Do virtual reality games improve mobility skills and balance measurements in community-dwelling older adults? Syst. Rev. Meta-Anal. Clin. Rehabil. 31(10), 1292–1304 (2017)CrossRefGoogle Scholar
  14. 14.
    D. Webster, O. Celik, Systematic review of kinect applications in elderly care and stroke rehabilitation. J. Neuroeng. Rehabil. 11(108), 1–24 (2014)Google Scholar
  15. 15.
    V.L. Klompstra, T. Jaarsma, A. Strömberg, Exergaming in older adults: a scoping review and implementation potential for patients with heart failure. Eur. J. Cardiovasc. Nurs. 13(5), 388–398 (2014)CrossRefGoogle Scholar
  16. 16.
    D.M. Harris, T. Rantalainen, M. Muthalib, L. Johnson, W.P. Teo, Exergaming as a viable therapeutic tool to improve static and dynamic balance among older adults and people with idiopathic Parkinson’s disease: a systematic review and meta-analysis. Front Aging Neurosci. 7, 167 (2015)Google Scholar
  17. 17.
    P.L. Weiss, E. Tirosh, D. Fehlings, Role of virtual reality for cerebral palsy management. J. Child Neurol. 29(8), 1119–1124 (2014)CrossRefGoogle Scholar
  18. 18.
    B. Bonnechere, B. Jansen, L. Omelina, M. Degelaen, V. Wermenbol, M. Rooze, S. Van Sint, Jan. Can serious games be incorporated with conventional treatment of children with cerebral palsy? A review. Res. Dev. Disabil. 35, 1899–1913 (2014)CrossRefGoogle Scholar
  19. 19.
    Y. Laufer, G. Dar, E. Kodeak, Does a Wii-based exercise program enhance balance control of independently functioning older adults? A systematic review. Clin. Interv. Aging 1803–1813 (2014)Google Scholar
  20. 20.
    S.D. Choi, L. Guo, D. Kang, S. Xiong, Exergame technology and interactive interventions for elderly fall prevention: a systematic review. Appl. Ergon. 1–12 (2016)Google Scholar
  21. 21.
    E. Pietrzak, C. Cotea, S. Pullman, Using commercial video games for falls prevention in older adults: the way for the future? J. Geriatr. Phys. Ther. 37, 166–177 (2014)CrossRefGoogle Scholar
  22. 22.
    K.I. Molina, N.A. Ricci, S.A. Morais, M. Rodrigues, Virtual Reality using games for improving physical functioning in older adults: a systematic review. J NeuroEng. Rehabil. 11(156), 1–20 (2014)Google Scholar
  23. 23.
    H.M. Hondori, M. Khademi, A review on technical and clinical impact of Microsoft Kinect on physical therapy and rehabilitation. J. Med. Eng. (Hindawi Publishing Corporation) 846514, 1–16 (2014)Google Scholar
  24. 24.
    S.R. Ellis, What are virtual environments. IEEE Comput. Gr. Appl. 14(1), 17–22 (1994)CrossRefGoogle Scholar
  25. 25.
    K.J. Miller, B.S. Adair, A.J. Pearce, C.M. Said, E. Ozanne, M.M. Morris, Effectiveness and feasibility of virtual reality and gaming system use at home by older adults for enabling physical activity to improve health-related domains: a systematic review. Age Ageing 43(2), 188–195 (2014)CrossRefGoogle Scholar
  26. 26.
    O. Wasenmüller, D. Stricker, Comparison of kinect v1 and v2 depth images in terms of accuracy and precision, in Asian Conference on Computer VisionACCV 2016 Workshops (2016), pp. 34–45Google Scholar
  27. 27.
    MS Press, PrimeSense supplies 3-D-sensing technology to “Project Natal” for Xbox 360 (MsPress, 31 March, 2010),
  28. 28.
    F. Weichert, D. Bachmann, B. Rudak, D. Fisseler, Analysis of the accuracy and robustness of the Leap Motion controller. Sensors 13, 6380–6393 (2013)CrossRefGoogle Scholar
  29. 29.
    J. Guna, G. Jakus, M. Pogacnik, S. Tomazic, J. Sodnik, Na analysis of the precision and reliability of the Leap Motion sensor and its suitability for static and dynamic tracking. Sensors 14, 3702–3720 (2014)CrossRefGoogle Scholar
  30. 30.
    H. Smeragliuolo, N.J. Hill, L. Disla, D. Putrino, Validation of the Leap Motion controller using markered motion capture technology. J. Biomech. 49, 1742–1750 (2016)CrossRefGoogle Scholar
  31. 31.
  32. 32.
    K.H. Chen, P.C. Lin, Y.J. Chen, B.S. Yang, C.H. Lin, Development of method for quantifying essential tremor using a small optical device. J. Neurosci. Method 266, 78–83 (2016)CrossRefGoogle Scholar
  33. 33.
    J.M. de Oliveira, R.C.G. Fernandes, C.S. Pinto, P.R. Pinheiro, S. Ribeiro, V.H.C. de Albuquerque. Novel virtual environment for alternative treatment of children with cerebral palsy. Comput. Intell. Neurosci. 8984379 (2016)Google Scholar
  34. 34.
    M. Iosa, G. Morone, A. Fusco, M. Castagnoli, F.R. Fusco, L. Pratesi, S. Paolucci. Leap Motion controlled videogame-based therapy for rehabilitation of elderly patients with subacute stroke: a feasibility pilot study. Top Stroke Rehabil. 306–316 (2015)Google Scholar
  35. 35.
  36. 36.
    D.J. Bowen, M. Kreuter, B. Spring, L. Cofta-Woerpel, L. Linnan, D. Weiner, S. Bakken, C.P. Kaplan, L. Squiers, C. Fabrizio, M. Fernandez, How we design feasibility studies. Am. J. Prev. Med. 36(5), 452–457 (2009)CrossRefGoogle Scholar
  37. 37.
    M. van Diest, C.C. Lamoth, J. Stegenga, G.J. Verkerke, K. Postema, Exergaming for balance training of elderly: state of the art and future developments. J Neuroeng. Rehabil. Engl. BioMed. Central 10, 101 (2013)CrossRefGoogle Scholar
  38. 38.
    C.M. Bleakley, D. Charles, A. Porter-Armstrong, M.D.J. McNeill, S.M. McDonough, B. McCormack. Gaming for Health: a systematic review of the physical and cognitive effects of interactive video games in older adults. J. Appl. Gerontol. 34, NP166–NP189 (2015)Google Scholar
  39. 39.
    D.C. Ribeiro-Papa, T. Massetti, T.B. Crocetta, L.D.C. Menezes, T.P.C. Antunes, I.M.P. Bezerra, C.B.M. Monteiro. Motor learning through virtual reality in elderly—a systematic review. Med Express 3 (2016)Google Scholar
  40. 40.
    N. Zeng, Z. Pope, J.E. Lee, Z. Gao, A systematic review of active video games on rehabilitative outcomes among older patients. J. Sport Heal. Sci. 6, 33–43 (2017)CrossRefGoogle Scholar
  41. 41.
    A. Nawaz, N. Skjaeret, J.L. Helbostad, Usability and acceptability of balance exergames in older adults: a scoping review. Health Inf. J. 22(4), 911–931 (2016)CrossRefGoogle Scholar
  42. 42.
    F.W. Simor, M.R. Brum, J.D.E. Schmidt, R. Rieder, A.C.B. de Marchi, Usability evaluation methods for gesture-based games: a systematic review. JMIR Serious Games 4(2), e17 (2016)CrossRefGoogle Scholar
  43. 43.
    D. Kairy, P. Lehoux, C. Vincent, M. Visitin, A systematic review of clinical outcomes, clinical process, healthcare utilization and costs associated with telerehabilitation. Disabil. Rehabil. 31, 427–447 (2009)Google Scholar
  44. 44.
    J.E. Maddux, Self-efficacy Theory (Springer, New York, 1995)CrossRefGoogle Scholar
  45. 45.
    V. Venkatesh, M.G. Morris, G.B. Davis, F.D. Davis, User acceptance of information technology: toward a unified view. MIS Q. 27(3), 425–478 (2003)CrossRefGoogle Scholar
  46. 46.
    F.D. Davis, Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 13, 319–340 (1989)CrossRefGoogle Scholar
  47. 47.
    Ajzen, The theory of planned behavior. Organ Behav. Hum. Decis. Process 50(2), 179–211 (1991)Google Scholar
  48. 48.
    R.P. Bagozzi, The legacy of the technology acceptance model and a proposal for a paradigm shift. J. Assoc. Inf. Syst. 8, 244–254 (2007)Google Scholar
  49. 49.
    J. Nielsen, Usability Engineering (Elsevier, Amsterdam, 1994)zbMATHGoogle Scholar
  50. 50.
    B. Shneiderman, Designing the User Interface: Strategies for Effective Human-Computer Interaction (Pearson Education India, New Delhi, 1986)Google Scholar
  51. 51.
    K.M. Gerling, J. Schild, M. Masuch, Exergaming for elderly: analyzing player experience and performance. Mensch Comput. (2011)Google Scholar
  52. 52.
    K.-K. Kang, J.-A. Kim, D. Kim, Development of a sensory gate–ball game system for the aged people. Vis. Comput. 25(12), 1073–1083 (2009)CrossRefGoogle Scholar
  53. 53.
    T.C. Chan, F. Chan, Y.F. Shea, O.Y. Lin, Y.K. Luk, F.H. Chan, Interactive virtual reality Wii in geriatric day hospital: a study to assess its feasibility, acceptability and efficacy. Geriatr. Gerontol. Int. 12(4), 714–721 (2012)CrossRefGoogle Scholar
  54. 54.
    Y.-Y. Chao, Y.K. Scherer, Y.-W. Wu, K.T. Lucke, C.A. Montgomery, The feasibility of an intervention combining self-efficacy theory and Wii Fit exergames in assisted living residents: a pilot study. Geriatr Nurs 34(5), 377–382 (2013)CrossRefGoogle Scholar
  55. 55.
    A.S. Billis, E.I. Konstantinidis, A.I. Ladas, M.N. Tsolaki, C. Pappas, P.D. Bamidis. Evaluating affective usability experiences of an exergaming platform for seniors, in Proceedings of 10th International Workshop on Biomedical Engineering, Kos, October 2011Google Scholar
  56. 56.
    J.M. Flach, C.O. Dominguez, User-centered design: integrating the user, instrument, and goal. Ergon Des. 19–24 (1995)Google Scholar
  57. 57.
    K. Isbister, N. Shaffer, Heuristic evaluation of games, in Game Usability: Advancing the Player Experience (CRC Press, 2008), pp. 79–89Google Scholar
  58. 58.
    R. Bernhaupt, Evaluating user experience in games: concepts and methods, in Human Computer Interaction Series (2010)Google Scholar
  59. 59.
    P. Sweetser, P. Wyeth, GameFlow: a model for evaluating player enjoyment in games. Comput. Entertain. 3, 3 (2005)CrossRefGoogle Scholar
  60. 60.
    P. Bonato, Advances in wearable technology and applications in physical medicine and rehabilitation. J. Neuroeng. Rehabil. 2, 2 (2005)CrossRefGoogle Scholar
  61. 61.
    S. Obdrzalek, G. Kurillo, F. Ofli, R. Bajcsy E. Seto, H. Jimison, M. Pavel. Accuracy and robustness of kinect pose estimation in the context of coaching of elderly population, in Proceedings of IEEE Engineering in Medicine and Biology Society (EMBC) (2012), pp. 1188–1193Google Scholar
  62. 62.
    G.C. Burdea, Virtual rehabilitation—benefits and challenges. Methods Inf. Med. 42(519), 1–11 (2003)Google Scholar
  63. 63.
    M. Kim, C. Jeon, J. Kim. A study on immersion and presence of a portable hand haptic system for immersive virtual reality. Sensors, 17(5), pii: E1141 (2017)Google Scholar
  64. 64.
    B.G. Witmer, M.J. Singer, Measuring presence in virtual environments: a presence questionnaire. Presence 7, 225–240 (1998)CrossRefGoogle Scholar
  65. 65.
    D. Corbetta, F. Imeri, R. Gatti, Rehabilitation that incorporates virtual reality is more effective than standard rehabilitation for improving walking speed, balance and mobility after stroke: a systematic review. J. Physiother. 61, 117–124 (2015)CrossRefGoogle Scholar
  66. 66.
    N.A. Borghese, M. Pirovano, P.L. Lanzi, S. Wuest, E.D. de Bruin, Computational intelligence and game design for effective at-home stroke rehabilitation. Games Health J. Res. Dev. Clin. Appl. 2(2), 81–88 (2013)Google Scholar
  67. 67.
    B. Lange, C.Y. Chang, E. Suma, B. Newman, A.S. Rizzo, M. Bolas. Development and evaluation of low cost game-based balance rehabilitation tool using the microsoft kinect sensor, in Proceedings of IEEE Engineering in Medicine and Biology Society, EMBC (2011), pp. 1831–1834Google Scholar
  68. 68.
    R. Hunicke, M. Le Blanc, R. Zubek, MDA: a formal approach to game design and game research, in Proceedings of AAAI-04 Workshop on Challenges in Game AI, July 2004, pp. 1–5Google Scholar
  69. 69.
    R. Dillon. The 6-11 framework: a new methodology for game analysis and design, in Proceedings of GAMEON, Singapore, Asia, March 2011, pp. 25–29Google Scholar
  70. 70.
    V. Stavljanin, M. Minovic, Gamification ecosystems: current state and perspectives, in Open Source Solutions for Knowledge Management and Technological Ecosystems (IGI Global, 2017)Google Scholar
  71. 71.
    G. Zichermann, C. Cunningham, Gamification by Design. Implementing Game Mechanics in Web and Mobile Apps (O’Reilly Media, Sebastopol, C.A., 2011)Google Scholar
  72. 72.
    K. Werbach, D. Hunter. For the Win. How Game Thinking Can Revolutionize Your Business. (Wharton Digital Press, 2012)Google Scholar
  73. 73.
  74. 74.
    M. Pirovano, E. Surer, R. Mainetti, P.L. Lanzi, N.A. Borghese, Exergaming and rehabilitation: a methodology for the design of effective and safe therapeutic exergames. Entertain. Comput. 14, 55–65 (2016)CrossRefGoogle Scholar
  75. 75.
    Y-L. Theng, A.B. Dahlan, M.L. Akmal, T.Z. Myint. An exploratory study on senior citizens’ perceptions of the Nintento Wii: the case of Singapore, in Proceedings of 3rd International Convention on Rehabilitation Engineering &Assistive Technology, April, Singapore (2009)Google Scholar
  76. 76.
    B. Bonnechere, B. Jansen, L. Omelina, S. Van Sint, Jan. The use of commercial video games in rehabilitation: a systematic review. Int. J. Rehabil. Res. 39(4), 277–290 (2016)CrossRefGoogle Scholar
  77. 77.
    D. Collado-Mateo, E. Merellano-Navarro, P.R. Olivares, J. Garcia-Rubio, N. Gusi, Effect of exergames on musculoskeletal pain. a systemeatic review and meta-analysis. Scand. J. Med. Sci. Sports 28, 760–771 (2018)CrossRefGoogle Scholar
  78. 78.
    G. Tieri, G. Morone, S. Paolucci, M. Iosa, Virtual reality in cognitive and motor rehabilitation: facts, fiction and fallacies. Expert Rev. Med. Devices 15(2), 107–117 (2018)CrossRefGoogle Scholar
  79. 79.
    R. Aarhus, E. Grönvall, S.B. Larsen, S. Wollsen, Turning training into play: embodied gaming, seniors, physical training and motivation. Gerontechnology 10(2), 110–120 (2011)CrossRefGoogle Scholar
  80. 80.
    M.P.J. Habgood, S.E. Ainsworth, Motivating children to learn effectively: exploring the value of intrinsic integration in educational games. J. Learn. Sci. 20(2), 169–206 (2011)CrossRefGoogle Scholar
  81. 81.
    J. Huberty, L. Ransdell, C. Sidman, Explaining long-term exercise adherence in women who complete a structured exercise program. Res. Q. Exerc. Sport 79, 374 (2008)CrossRefGoogle Scholar
  82. 82.
    S.R. Wood, N. Murillo, P. Bach-y-Rita P, R.S. Leder, J.T. Marks, S.J. Page, Motivating, game-based stroke rehabilitation: a brief report. Top Stroke Rehabil. 10, 134–140 (2003)Google Scholar
  83. 83.
    Y. Tian, Y. Bian, P. Han, P. Wang, F. Gao, Y. Chen, Physiological signal analysis for evaluating flow during playing of computer games of varying difficulty. Front Psychol. 8, 1121 (2017)CrossRefGoogle Scholar
  84. 84.
    R.M. Ryan, C.S. Rigby, A. Przybylski, The motivational pull of video games: a self-determination theory approach. Motiv. Emot. 30, 347–363 (2006)Google Scholar
  85. 85.
    R.M. Ryan, E.L. Deci, Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. Am. Psychol. 55, 68–78 (2000)CrossRefGoogle Scholar
  86. 86.
    S. Rigby, R. Richard. Immersion and presence, in Glued to Games: How Video Games Draw Us In and Hold Us Spellbound (2011), pp. 81–96Google Scholar
  87. 87.
    B.D. Sylvester, M. Standage, J. Dowd, L.J. Martin, S.N. Sweet, M.R. Beauchamp, Perceived variety, psychological needs satisfaction and exercise-related well-being. Psychol. Health 29(9), 1041–1061 (2014)CrossRefGoogle Scholar
  88. 88.
    A. Bandura, Social Foundations of Thought and Action: A Social Cognitive Theory (Prentice-Hall, Englewood Cliffs, N.J., 1986)Google Scholar
  89. 89.
    M.W. Kreuter, D.W. Farrell, L.R. Olevitch, L.K. Brennan. Tailoring Health Messages: Customizing Communication with Computer Technology (Lawrence Erlbaum Associates, 2000)Google Scholar
  90. 90.
    D. Thompson, J. Baranowski, K. Cullen, T. Baranowski, Development of a theory-based internet program promoting maintenance of diet and physical activity change to 8-year-old African American girls. Comput. Educ. 48, 446–459 (2007)CrossRefGoogle Scholar
  91. 91.
    L.S. Vygotsky, Mind in Society. The Development of Higher Psychological Processes (1978)Google Scholar
  92. 92.
    D. Shernoff, M. Csikszentmihalyi, B. Schneider, E. Shernoff, Student engagement in high school classrooms from the perspective of flow theory. Sch. Psychol. Q. 18, 156–176 (2003)CrossRefGoogle Scholar
  93. 93.
    R. Garris, R. Ahlers, J.E. Driskell. Games, motivation, and learning: a research and practice model. Simul. Gaming 441–467 (2002)Google Scholar
  94. 94.
    B.H. Dobkin, Rehabilitation after stroke. N. Engl. J. Med. 352(16), 1677–1684 (2005)CrossRefGoogle Scholar
  95. 95.
    I-T. Chiang, Old dogs can learn new tricks: exploring effective strategies to facilitate somatosensory video games for institutionalized older veterans, in Edutainment technologies. Educational games and virtual reality/augmented reality applications, ed. by M. Chang, W.-Y. Hwang, M.-P. Chen, et al. (Springer, Berlin)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Gabriela Postolache
    • 1
  • Francisco Carry
    • 1
  • Filipe Lourenço
    • 2
  • Diogo Ferreira
    • 2
  • Raul Oliveira
    • 3
  • Pedro Silva Girão
    • 1
  • Octavian Postolache
    • 1
    • 2
    Email author
  1. 1.Instituto de TelecomunicacoesLisbonPortugal
  2. 2.ISCTE-Instituto Universitario de LisboaLisbonPortugal
  3. 3.Faculdade de Motricidade HumanaULLisbonPortugal

Personalised recommendations