Skip to main content

Serious Games Based on Kinect and Leap Motion Controller for Upper Limbs Physical Rehabilitation

Part of the Smart Sensors, Measurement and Instrumentation book series (SSMI,volume 29)

Abstract

The design, implementation and tests of a system for assessment and monitoring movements, which includes the sensors from Kinect and Leap Motion Controller devices, are discussed in the present chapter. The advantages and some drawbacks of using the two devices for creating virtual environments for motor rehabilitation in which interaction of the user with virtual reality is made through natural user interfaces are described. The IoT architecture for rehabilitation environment, several serious games that our team have developed and the usability evaluation of the system are presented. Our insights, based on our research work during serious games development as well as on literature analysis, mainly focusing these on practicality of the developed serious games and on their acceptability for motor rehabilitation, are also included in the chapter. In the development of a system that includes IoT technology for movements tracking and progress evaluation during motor rehabilitation, the importance of user centered design is underscored.

Keywords

  • Kinect
  • Leap Motion
  • Virtual reality
  • Usability

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-99540-3_8
  • Chapter length: 31 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   139.00
Price excludes VAT (USA)
  • ISBN: 978-3-319-99540-3
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   179.99
Price excludes VAT (USA)
Hardcover Book
USD   179.99
Price excludes VAT (USA)
Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

References

  1. Game Market, Statica (2018), https://www.statica.com/statistics/246888/value-of-the-global-video-game-market/

  2. Pew Research Center—Number of Gamers. Statica (2018), https://www.statica.com/statistics/748044/number-video-gamers-world/

  3. Best Selling Games, Statica (2018), https://www.statista.com/statistics/264530/all-time-best-selling-console-games-worldwide/

  4. RnR, https://www.marketsandmarkets.com/PressReleases/serious-game.asp

  5. Game Industry Revenue. Statica (2018), https://www.statista.com/statistics/733616/game-based-learning-industry-revenue-world/

  6. C.C. Abt, Serious Games (University Press of America, New York, 1970)

    Google Scholar 

  7. B.P. Bergeron. Developing Serious Games, 1st edn. (Hingham Charles River Media, 2006)

    Google Scholar 

  8. D. Michael, S. Chen, Serious Games that Educate, Train and Inform (Thomson, Boston, MA, 2006)

    Google Scholar 

  9. D. Thompson, T. Baranowski, R. Buday. Serious video games for health. How behavioral science guided the development of a serious video game. Simul Gaming 41(4), 587–606 (2010)

    Google Scholar 

  10. D.M. Harris, T. Rantalainen, M. Muthalib, L. Johnson, W.-P. Teo, Exergaming as a viable therapeutic tool to improve static and dynamic balance among older adults and people with idiopathic Parkinson’s disease: a systematic review and meta-analysis. Front Aging Neurosci. 7(167), 1–12 (2015)

    Google Scholar 

  11. N. Skjæret, A. Nawaz, T. Morat, D. Schoene, J.L. Helbostad, B. Vereijken, Exercise and rehabilitation delivered through exergames in older adults: an integrative review of technologies, safety and efficacy. Int. J. Med. Inform. 85(1), 1–16 (2016)

    CrossRef  Google Scholar 

  12. L. Donath, R. Rössler, O. Faude, Effects of virtual reality training (exergaming) compared to alternative exercise training and passive control on standing balance and functional mobility in healthy community-dwelling seniors: a meta-analytical review. Sports Med. 46(9), 1293–1309 (2016)

    CrossRef  Google Scholar 

  13. S.G.R. Neri, J.R. Cardoso, L. Cruz, R.M. Lima, R.J. de Oliveira, M.D. Iversen, R.L. Carregaro, Do virtual reality games improve mobility skills and balance measurements in community-dwelling older adults? Syst. Rev. Meta-Anal. Clin. Rehabil. 31(10), 1292–1304 (2017)

    CrossRef  Google Scholar 

  14. D. Webster, O. Celik, Systematic review of kinect applications in elderly care and stroke rehabilitation. J. Neuroeng. Rehabil. 11(108), 1–24 (2014)

    Google Scholar 

  15. V.L. Klompstra, T. Jaarsma, A. Strömberg, Exergaming in older adults: a scoping review and implementation potential for patients with heart failure. Eur. J. Cardiovasc. Nurs. 13(5), 388–398 (2014)

    CrossRef  Google Scholar 

  16. D.M. Harris, T. Rantalainen, M. Muthalib, L. Johnson, W.P. Teo, Exergaming as a viable therapeutic tool to improve static and dynamic balance among older adults and people with idiopathic Parkinson’s disease: a systematic review and meta-analysis. Front Aging Neurosci. 7, 167 (2015)

    Google Scholar 

  17. P.L. Weiss, E. Tirosh, D. Fehlings, Role of virtual reality for cerebral palsy management. J. Child Neurol. 29(8), 1119–1124 (2014)

    CrossRef  Google Scholar 

  18. B. Bonnechere, B. Jansen, L. Omelina, M. Degelaen, V. Wermenbol, M. Rooze, S. Van Sint, Jan. Can serious games be incorporated with conventional treatment of children with cerebral palsy? A review. Res. Dev. Disabil. 35, 1899–1913 (2014)

    CrossRef  Google Scholar 

  19. Y. Laufer, G. Dar, E. Kodeak, Does a Wii-based exercise program enhance balance control of independently functioning older adults? A systematic review. Clin. Interv. Aging 1803–1813 (2014)

    Google Scholar 

  20. S.D. Choi, L. Guo, D. Kang, S. Xiong, Exergame technology and interactive interventions for elderly fall prevention: a systematic review. Appl. Ergon. 1–12 (2016)

    Google Scholar 

  21. E. Pietrzak, C. Cotea, S. Pullman, Using commercial video games for falls prevention in older adults: the way for the future? J. Geriatr. Phys. Ther. 37, 166–177 (2014)

    CrossRef  Google Scholar 

  22. K.I. Molina, N.A. Ricci, S.A. Morais, M. Rodrigues, Virtual Reality using games for improving physical functioning in older adults: a systematic review. J NeuroEng. Rehabil. 11(156), 1–20 (2014)

    Google Scholar 

  23. H.M. Hondori, M. Khademi, A review on technical and clinical impact of Microsoft Kinect on physical therapy and rehabilitation. J. Med. Eng. (Hindawi Publishing Corporation) 846514, 1–16 (2014)

    Google Scholar 

  24. S.R. Ellis, What are virtual environments. IEEE Comput. Gr. Appl. 14(1), 17–22 (1994)

    CrossRef  Google Scholar 

  25. K.J. Miller, B.S. Adair, A.J. Pearce, C.M. Said, E. Ozanne, M.M. Morris, Effectiveness and feasibility of virtual reality and gaming system use at home by older adults for enabling physical activity to improve health-related domains: a systematic review. Age Ageing 43(2), 188–195 (2014)

    CrossRef  Google Scholar 

  26. O. Wasenmüller, D. Stricker, Comparison of kinect v1 and v2 depth images in terms of accuracy and precision, in Asian Conference on Computer VisionACCV 2016 Workshops (2016), pp. 34–45

    Google Scholar 

  27. MS Press, PrimeSense supplies 3-D-sensing technology to “Project Natal” for Xbox 360 (MsPress, 31 March, 2010), https://news.microsoft.com/2010/03/31/primesense-supplies-3-d-sensing-technology-to-project-natal-for-xbox-360/

  28. F. Weichert, D. Bachmann, B. Rudak, D. Fisseler, Analysis of the accuracy and robustness of the Leap Motion controller. Sensors 13, 6380–6393 (2013)

    CrossRef  Google Scholar 

  29. J. Guna, G. Jakus, M. Pogacnik, S. Tomazic, J. Sodnik, Na analysis of the precision and reliability of the Leap Motion sensor and its suitability for static and dynamic tracking. Sensors 14, 3702–3720 (2014)

    CrossRef  Google Scholar 

  30. H. Smeragliuolo, N.J. Hill, L. Disla, D. Putrino, Validation of the Leap Motion controller using markered motion capture technology. J. Biomech. 49, 1742–1750 (2016)

    CrossRef  Google Scholar 

  31. Leap Motion, https://www.leapmotion.com/

  32. K.H. Chen, P.C. Lin, Y.J. Chen, B.S. Yang, C.H. Lin, Development of method for quantifying essential tremor using a small optical device. J. Neurosci. Method 266, 78–83 (2016)

    CrossRef  Google Scholar 

  33. J.M. de Oliveira, R.C.G. Fernandes, C.S. Pinto, P.R. Pinheiro, S. Ribeiro, V.H.C. de Albuquerque. Novel virtual environment for alternative treatment of children with cerebral palsy. Comput. Intell. Neurosci. 8984379 (2016)

    Google Scholar 

  34. M. Iosa, G. Morone, A. Fusco, M. Castagnoli, F.R. Fusco, L. Pratesi, S. Paolucci. Leap Motion controlled videogame-based therapy for rehabilitation of elderly patients with subacute stroke: a feasibility pilot study. Top Stroke Rehabil. 306–316 (2015)

    Google Scholar 

  35. VirtualWare, virtualwaregroup.com

  36. D.J. Bowen, M. Kreuter, B. Spring, L. Cofta-Woerpel, L. Linnan, D. Weiner, S. Bakken, C.P. Kaplan, L. Squiers, C. Fabrizio, M. Fernandez, How we design feasibility studies. Am. J. Prev. Med. 36(5), 452–457 (2009)

    CrossRef  Google Scholar 

  37. M. van Diest, C.C. Lamoth, J. Stegenga, G.J. Verkerke, K. Postema, Exergaming for balance training of elderly: state of the art and future developments. J Neuroeng. Rehabil. Engl. BioMed. Central 10, 101 (2013)

    CrossRef  Google Scholar 

  38. C.M. Bleakley, D. Charles, A. Porter-Armstrong, M.D.J. McNeill, S.M. McDonough, B. McCormack. Gaming for Health: a systematic review of the physical and cognitive effects of interactive video games in older adults. J. Appl. Gerontol. 34, NP166–NP189 (2015)

    Google Scholar 

  39. D.C. Ribeiro-Papa, T. Massetti, T.B. Crocetta, L.D.C. Menezes, T.P.C. Antunes, I.M.P. Bezerra, C.B.M. Monteiro. Motor learning through virtual reality in elderly—a systematic review. Med Express 3 (2016)

    Google Scholar 

  40. N. Zeng, Z. Pope, J.E. Lee, Z. Gao, A systematic review of active video games on rehabilitative outcomes among older patients. J. Sport Heal. Sci. 6, 33–43 (2017)

    CrossRef  Google Scholar 

  41. A. Nawaz, N. Skjaeret, J.L. Helbostad, Usability and acceptability of balance exergames in older adults: a scoping review. Health Inf. J. 22(4), 911–931 (2016)

    CrossRef  Google Scholar 

  42. F.W. Simor, M.R. Brum, J.D.E. Schmidt, R. Rieder, A.C.B. de Marchi, Usability evaluation methods for gesture-based games: a systematic review. JMIR Serious Games 4(2), e17 (2016)

    CrossRef  Google Scholar 

  43. D. Kairy, P. Lehoux, C. Vincent, M. Visitin, A systematic review of clinical outcomes, clinical process, healthcare utilization and costs associated with telerehabilitation. Disabil. Rehabil. 31, 427–447 (2009)

    Google Scholar 

  44. J.E. Maddux, Self-efficacy Theory (Springer, New York, 1995)

    CrossRef  Google Scholar 

  45. V. Venkatesh, M.G. Morris, G.B. Davis, F.D. Davis, User acceptance of information technology: toward a unified view. MIS Q. 27(3), 425–478 (2003)

    CrossRef  Google Scholar 

  46. F.D. Davis, Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 13, 319–340 (1989)

    CrossRef  Google Scholar 

  47. Ajzen, The theory of planned behavior. Organ Behav. Hum. Decis. Process 50(2), 179–211 (1991)

    Google Scholar 

  48. R.P. Bagozzi, The legacy of the technology acceptance model and a proposal for a paradigm shift. J. Assoc. Inf. Syst. 8, 244–254 (2007)

    Google Scholar 

  49. J. Nielsen, Usability Engineering (Elsevier, Amsterdam, 1994)

    MATH  Google Scholar 

  50. B. Shneiderman, Designing the User Interface: Strategies for Effective Human-Computer Interaction (Pearson Education India, New Delhi, 1986)

    Google Scholar 

  51. K.M. Gerling, J. Schild, M. Masuch, Exergaming for elderly: analyzing player experience and performance. Mensch Comput. (2011)

    Google Scholar 

  52. K.-K. Kang, J.-A. Kim, D. Kim, Development of a sensory gate–ball game system for the aged people. Vis. Comput. 25(12), 1073–1083 (2009)

    CrossRef  Google Scholar 

  53. T.C. Chan, F. Chan, Y.F. Shea, O.Y. Lin, Y.K. Luk, F.H. Chan, Interactive virtual reality Wii in geriatric day hospital: a study to assess its feasibility, acceptability and efficacy. Geriatr. Gerontol. Int. 12(4), 714–721 (2012)

    CrossRef  Google Scholar 

  54. Y.-Y. Chao, Y.K. Scherer, Y.-W. Wu, K.T. Lucke, C.A. Montgomery, The feasibility of an intervention combining self-efficacy theory and Wii Fit exergames in assisted living residents: a pilot study. Geriatr Nurs 34(5), 377–382 (2013)

    CrossRef  Google Scholar 

  55. A.S. Billis, E.I. Konstantinidis, A.I. Ladas, M.N. Tsolaki, C. Pappas, P.D. Bamidis. Evaluating affective usability experiences of an exergaming platform for seniors, in Proceedings of 10th International Workshop on Biomedical Engineering, Kos, October 2011

    Google Scholar 

  56. J.M. Flach, C.O. Dominguez, User-centered design: integrating the user, instrument, and goal. Ergon Des. 19–24 (1995)

    Google Scholar 

  57. K. Isbister, N. Shaffer, Heuristic evaluation of games, in Game Usability: Advancing the Player Experience (CRC Press, 2008), pp. 79–89

    Google Scholar 

  58. R. Bernhaupt, Evaluating user experience in games: concepts and methods, in Human Computer Interaction Series (2010)

    Google Scholar 

  59. P. Sweetser, P. Wyeth, GameFlow: a model for evaluating player enjoyment in games. Comput. Entertain. 3, 3 (2005)

    CrossRef  Google Scholar 

  60. P. Bonato, Advances in wearable technology and applications in physical medicine and rehabilitation. J. Neuroeng. Rehabil. 2, 2 (2005)

    CrossRef  Google Scholar 

  61. S. Obdrzalek, G. Kurillo, F. Ofli, R. Bajcsy E. Seto, H. Jimison, M. Pavel. Accuracy and robustness of kinect pose estimation in the context of coaching of elderly population, in Proceedings of IEEE Engineering in Medicine and Biology Society (EMBC) (2012), pp. 1188–1193

    Google Scholar 

  62. G.C. Burdea, Virtual rehabilitation—benefits and challenges. Methods Inf. Med. 42(519), 1–11 (2003)

    Google Scholar 

  63. M. Kim, C. Jeon, J. Kim. A study on immersion and presence of a portable hand haptic system for immersive virtual reality. Sensors, 17(5), pii: E1141 (2017)

    Google Scholar 

  64. B.G. Witmer, M.J. Singer, Measuring presence in virtual environments: a presence questionnaire. Presence 7, 225–240 (1998)

    CrossRef  Google Scholar 

  65. D. Corbetta, F. Imeri, R. Gatti, Rehabilitation that incorporates virtual reality is more effective than standard rehabilitation for improving walking speed, balance and mobility after stroke: a systematic review. J. Physiother. 61, 117–124 (2015)

    CrossRef  Google Scholar 

  66. N.A. Borghese, M. Pirovano, P.L. Lanzi, S. Wuest, E.D. de Bruin, Computational intelligence and game design for effective at-home stroke rehabilitation. Games Health J. Res. Dev. Clin. Appl. 2(2), 81–88 (2013)

    Google Scholar 

  67. B. Lange, C.Y. Chang, E. Suma, B. Newman, A.S. Rizzo, M. Bolas. Development and evaluation of low cost game-based balance rehabilitation tool using the microsoft kinect sensor, in Proceedings of IEEE Engineering in Medicine and Biology Society, EMBC (2011), pp. 1831–1834

    Google Scholar 

  68. R. Hunicke, M. Le Blanc, R. Zubek, MDA: a formal approach to game design and game research, in Proceedings of AAAI-04 Workshop on Challenges in Game AI, July 2004, pp. 1–5

    Google Scholar 

  69. R. Dillon. The 6-11 framework: a new methodology for game analysis and design, in Proceedings of GAMEON, Singapore, Asia, March 2011, pp. 25–29

    Google Scholar 

  70. V. Stavljanin, M. Minovic, Gamification ecosystems: current state and perspectives, in Open Source Solutions for Knowledge Management and Technological Ecosystems (IGI Global, 2017)

    Google Scholar 

  71. G. Zichermann, C. Cunningham, Gamification by Design. Implementing Game Mechanics in Web and Mobile Apps (O’Reilly Media, Sebastopol, C.A., 2011)

    Google Scholar 

  72. K. Werbach, D. Hunter. For the Win. How Game Thinking Can Revolutionize Your Business. (Wharton Digital Press, 2012)

    Google Scholar 

  73. Dev.Mag. devmag.org.za/2012/04/19/video-game-audio-diegesis-theory-2/

  74. M. Pirovano, E. Surer, R. Mainetti, P.L. Lanzi, N.A. Borghese, Exergaming and rehabilitation: a methodology for the design of effective and safe therapeutic exergames. Entertain. Comput. 14, 55–65 (2016)

    CrossRef  Google Scholar 

  75. Y-L. Theng, A.B. Dahlan, M.L. Akmal, T.Z. Myint. An exploratory study on senior citizens’ perceptions of the Nintento Wii: the case of Singapore, in Proceedings of 3rd International Convention on Rehabilitation Engineering &Assistive Technology, April, Singapore (2009)

    Google Scholar 

  76. B. Bonnechere, B. Jansen, L. Omelina, S. Van Sint, Jan. The use of commercial video games in rehabilitation: a systematic review. Int. J. Rehabil. Res. 39(4), 277–290 (2016)

    CrossRef  Google Scholar 

  77. D. Collado-Mateo, E. Merellano-Navarro, P.R. Olivares, J. Garcia-Rubio, N. Gusi, Effect of exergames on musculoskeletal pain. a systemeatic review and meta-analysis. Scand. J. Med. Sci. Sports 28, 760–771 (2018)

    CrossRef  Google Scholar 

  78. G. Tieri, G. Morone, S. Paolucci, M. Iosa, Virtual reality in cognitive and motor rehabilitation: facts, fiction and fallacies. Expert Rev. Med. Devices 15(2), 107–117 (2018)

    CrossRef  Google Scholar 

  79. R. Aarhus, E. Grönvall, S.B. Larsen, S. Wollsen, Turning training into play: embodied gaming, seniors, physical training and motivation. Gerontechnology 10(2), 110–120 (2011)

    CrossRef  Google Scholar 

  80. M.P.J. Habgood, S.E. Ainsworth, Motivating children to learn effectively: exploring the value of intrinsic integration in educational games. J. Learn. Sci. 20(2), 169–206 (2011)

    CrossRef  Google Scholar 

  81. J. Huberty, L. Ransdell, C. Sidman, Explaining long-term exercise adherence in women who complete a structured exercise program. Res. Q. Exerc. Sport 79, 374 (2008)

    CrossRef  Google Scholar 

  82. S.R. Wood, N. Murillo, P. Bach-y-Rita P, R.S. Leder, J.T. Marks, S.J. Page, Motivating, game-based stroke rehabilitation: a brief report. Top Stroke Rehabil. 10, 134–140 (2003)

    Google Scholar 

  83. Y. Tian, Y. Bian, P. Han, P. Wang, F. Gao, Y. Chen, Physiological signal analysis for evaluating flow during playing of computer games of varying difficulty. Front Psychol. 8, 1121 (2017)

    CrossRef  Google Scholar 

  84. R.M. Ryan, C.S. Rigby, A. Przybylski, The motivational pull of video games: a self-determination theory approach. Motiv. Emot. 30, 347–363 (2006)

    Google Scholar 

  85. R.M. Ryan, E.L. Deci, Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. Am. Psychol. 55, 68–78 (2000)

    CrossRef  Google Scholar 

  86. S. Rigby, R. Richard. Immersion and presence, in Glued to Games: How Video Games Draw Us In and Hold Us Spellbound (2011), pp. 81–96

    Google Scholar 

  87. B.D. Sylvester, M. Standage, J. Dowd, L.J. Martin, S.N. Sweet, M.R. Beauchamp, Perceived variety, psychological needs satisfaction and exercise-related well-being. Psychol. Health 29(9), 1041–1061 (2014)

    CrossRef  Google Scholar 

  88. A. Bandura, Social Foundations of Thought and Action: A Social Cognitive Theory (Prentice-Hall, Englewood Cliffs, N.J., 1986)

    Google Scholar 

  89. M.W. Kreuter, D.W. Farrell, L.R. Olevitch, L.K. Brennan. Tailoring Health Messages: Customizing Communication with Computer Technology (Lawrence Erlbaum Associates, 2000)

    Google Scholar 

  90. D. Thompson, J. Baranowski, K. Cullen, T. Baranowski, Development of a theory-based internet program promoting maintenance of diet and physical activity change to 8-year-old African American girls. Comput. Educ. 48, 446–459 (2007)

    CrossRef  Google Scholar 

  91. L.S. Vygotsky, Mind in Society. The Development of Higher Psychological Processes (1978)

    Google Scholar 

  92. D. Shernoff, M. Csikszentmihalyi, B. Schneider, E. Shernoff, Student engagement in high school classrooms from the perspective of flow theory. Sch. Psychol. Q. 18, 156–176 (2003)

    CrossRef  Google Scholar 

  93. R. Garris, R. Ahlers, J.E. Driskell. Games, motivation, and learning: a research and practice model. Simul. Gaming 441–467 (2002)

    Google Scholar 

  94. B.H. Dobkin, Rehabilitation after stroke. N. Engl. J. Med. 352(16), 1677–1684 (2005)

    CrossRef  Google Scholar 

  95. I-T. Chiang, Old dogs can learn new tricks: exploring effective strategies to facilitate somatosensory video games for institutionalized older veterans, in Edutainment technologies. Educational games and virtual reality/augmented reality applications, ed. by M. Chang, W.-Y. Hwang, M.-P. Chen, et al. (Springer, Berlin)

    Google Scholar 

Download references

Acknowledgements

The work was supported by Fundação para a Ciência e a Tecnologia, project PTDC/DTT-DES/6776/2014 and Instituto de Telecomunicações.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Octavian Postolache .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Verify currency and authenticity via CrossMark

Cite this chapter

Postolache, G. et al. (2019). Serious Games Based on Kinect and Leap Motion Controller for Upper Limbs Physical Rehabilitation. In: Mukhopadhyay, S., Jayasundera, K., Postolache, O. (eds) Modern Sensing Technologies . Smart Sensors, Measurement and Instrumentation, vol 29. Springer, Cham. https://doi.org/10.1007/978-3-319-99540-3_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-99540-3_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-99539-7

  • Online ISBN: 978-3-319-99540-3

  • eBook Packages: EngineeringEngineering (R0)