Abstract
The design, implementation and tests of a system for assessment and monitoring movements, which includes the sensors from Kinect and Leap Motion Controller devices, are discussed in the present chapter. The advantages and some drawbacks of using the two devices for creating virtual environments for motor rehabilitation in which interaction of the user with virtual reality is made through natural user interfaces are described. The IoT architecture for rehabilitation environment, several serious games that our team have developed and the usability evaluation of the system are presented. Our insights, based on our research work during serious games development as well as on literature analysis, mainly focusing these on practicality of the developed serious games and on their acceptability for motor rehabilitation, are also included in the chapter. In the development of a system that includes IoT technology for movements tracking and progress evaluation during motor rehabilitation, the importance of user centered design is underscored.
Keywords
- Kinect
- Leap Motion
- Virtual reality
- Usability
This is a preview of subscription content, access via your institution.
Buying options




















References
Game Market, Statica (2018), https://www.statica.com/statistics/246888/value-of-the-global-video-game-market/
Pew Research Center—Number of Gamers. Statica (2018), https://www.statica.com/statistics/748044/number-video-gamers-world/
Best Selling Games, Statica (2018), https://www.statista.com/statistics/264530/all-time-best-selling-console-games-worldwide/
RnR, https://www.marketsandmarkets.com/PressReleases/serious-game.asp
Game Industry Revenue. Statica (2018), https://www.statista.com/statistics/733616/game-based-learning-industry-revenue-world/
C.C. Abt, Serious Games (University Press of America, New York, 1970)
B.P. Bergeron. Developing Serious Games, 1st edn. (Hingham Charles River Media, 2006)
D. Michael, S. Chen, Serious Games that Educate, Train and Inform (Thomson, Boston, MA, 2006)
D. Thompson, T. Baranowski, R. Buday. Serious video games for health. How behavioral science guided the development of a serious video game. Simul Gaming 41(4), 587–606 (2010)
D.M. Harris, T. Rantalainen, M. Muthalib, L. Johnson, W.-P. Teo, Exergaming as a viable therapeutic tool to improve static and dynamic balance among older adults and people with idiopathic Parkinson’s disease: a systematic review and meta-analysis. Front Aging Neurosci. 7(167), 1–12 (2015)
N. Skjæret, A. Nawaz, T. Morat, D. Schoene, J.L. Helbostad, B. Vereijken, Exercise and rehabilitation delivered through exergames in older adults: an integrative review of technologies, safety and efficacy. Int. J. Med. Inform. 85(1), 1–16 (2016)
L. Donath, R. Rössler, O. Faude, Effects of virtual reality training (exergaming) compared to alternative exercise training and passive control on standing balance and functional mobility in healthy community-dwelling seniors: a meta-analytical review. Sports Med. 46(9), 1293–1309 (2016)
S.G.R. Neri, J.R. Cardoso, L. Cruz, R.M. Lima, R.J. de Oliveira, M.D. Iversen, R.L. Carregaro, Do virtual reality games improve mobility skills and balance measurements in community-dwelling older adults? Syst. Rev. Meta-Anal. Clin. Rehabil. 31(10), 1292–1304 (2017)
D. Webster, O. Celik, Systematic review of kinect applications in elderly care and stroke rehabilitation. J. Neuroeng. Rehabil. 11(108), 1–24 (2014)
V.L. Klompstra, T. Jaarsma, A. Strömberg, Exergaming in older adults: a scoping review and implementation potential for patients with heart failure. Eur. J. Cardiovasc. Nurs. 13(5), 388–398 (2014)
D.M. Harris, T. Rantalainen, M. Muthalib, L. Johnson, W.P. Teo, Exergaming as a viable therapeutic tool to improve static and dynamic balance among older adults and people with idiopathic Parkinson’s disease: a systematic review and meta-analysis. Front Aging Neurosci. 7, 167 (2015)
P.L. Weiss, E. Tirosh, D. Fehlings, Role of virtual reality for cerebral palsy management. J. Child Neurol. 29(8), 1119–1124 (2014)
B. Bonnechere, B. Jansen, L. Omelina, M. Degelaen, V. Wermenbol, M. Rooze, S. Van Sint, Jan. Can serious games be incorporated with conventional treatment of children with cerebral palsy? A review. Res. Dev. Disabil. 35, 1899–1913 (2014)
Y. Laufer, G. Dar, E. Kodeak, Does a Wii-based exercise program enhance balance control of independently functioning older adults? A systematic review. Clin. Interv. Aging 1803–1813 (2014)
S.D. Choi, L. Guo, D. Kang, S. Xiong, Exergame technology and interactive interventions for elderly fall prevention: a systematic review. Appl. Ergon. 1–12 (2016)
E. Pietrzak, C. Cotea, S. Pullman, Using commercial video games for falls prevention in older adults: the way for the future? J. Geriatr. Phys. Ther. 37, 166–177 (2014)
K.I. Molina, N.A. Ricci, S.A. Morais, M. Rodrigues, Virtual Reality using games for improving physical functioning in older adults: a systematic review. J NeuroEng. Rehabil. 11(156), 1–20 (2014)
H.M. Hondori, M. Khademi, A review on technical and clinical impact of Microsoft Kinect on physical therapy and rehabilitation. J. Med. Eng. (Hindawi Publishing Corporation) 846514, 1–16 (2014)
S.R. Ellis, What are virtual environments. IEEE Comput. Gr. Appl. 14(1), 17–22 (1994)
K.J. Miller, B.S. Adair, A.J. Pearce, C.M. Said, E. Ozanne, M.M. Morris, Effectiveness and feasibility of virtual reality and gaming system use at home by older adults for enabling physical activity to improve health-related domains: a systematic review. Age Ageing 43(2), 188–195 (2014)
O. Wasenmüller, D. Stricker, Comparison of kinect v1 and v2 depth images in terms of accuracy and precision, in Asian Conference on Computer Vision—ACCV 2016 Workshops (2016), pp. 34–45
MS Press, PrimeSense supplies 3-D-sensing technology to “Project Natal” for Xbox 360 (MsPress, 31 March, 2010), https://news.microsoft.com/2010/03/31/primesense-supplies-3-d-sensing-technology-to-project-natal-for-xbox-360/
F. Weichert, D. Bachmann, B. Rudak, D. Fisseler, Analysis of the accuracy and robustness of the Leap Motion controller. Sensors 13, 6380–6393 (2013)
J. Guna, G. Jakus, M. Pogacnik, S. Tomazic, J. Sodnik, Na analysis of the precision and reliability of the Leap Motion sensor and its suitability for static and dynamic tracking. Sensors 14, 3702–3720 (2014)
H. Smeragliuolo, N.J. Hill, L. Disla, D. Putrino, Validation of the Leap Motion controller using markered motion capture technology. J. Biomech. 49, 1742–1750 (2016)
Leap Motion, https://www.leapmotion.com/
K.H. Chen, P.C. Lin, Y.J. Chen, B.S. Yang, C.H. Lin, Development of method for quantifying essential tremor using a small optical device. J. Neurosci. Method 266, 78–83 (2016)
J.M. de Oliveira, R.C.G. Fernandes, C.S. Pinto, P.R. Pinheiro, S. Ribeiro, V.H.C. de Albuquerque. Novel virtual environment for alternative treatment of children with cerebral palsy. Comput. Intell. Neurosci. 8984379 (2016)
M. Iosa, G. Morone, A. Fusco, M. Castagnoli, F.R. Fusco, L. Pratesi, S. Paolucci. Leap Motion controlled videogame-based therapy for rehabilitation of elderly patients with subacute stroke: a feasibility pilot study. Top Stroke Rehabil. 306–316 (2015)
VirtualWare, virtualwaregroup.com
D.J. Bowen, M. Kreuter, B. Spring, L. Cofta-Woerpel, L. Linnan, D. Weiner, S. Bakken, C.P. Kaplan, L. Squiers, C. Fabrizio, M. Fernandez, How we design feasibility studies. Am. J. Prev. Med. 36(5), 452–457 (2009)
M. van Diest, C.C. Lamoth, J. Stegenga, G.J. Verkerke, K. Postema, Exergaming for balance training of elderly: state of the art and future developments. J Neuroeng. Rehabil. Engl. BioMed. Central 10, 101 (2013)
C.M. Bleakley, D. Charles, A. Porter-Armstrong, M.D.J. McNeill, S.M. McDonough, B. McCormack. Gaming for Health: a systematic review of the physical and cognitive effects of interactive video games in older adults. J. Appl. Gerontol. 34, NP166–NP189 (2015)
D.C. Ribeiro-Papa, T. Massetti, T.B. Crocetta, L.D.C. Menezes, T.P.C. Antunes, I.M.P. Bezerra, C.B.M. Monteiro. Motor learning through virtual reality in elderly—a systematic review. Med Express 3 (2016)
N. Zeng, Z. Pope, J.E. Lee, Z. Gao, A systematic review of active video games on rehabilitative outcomes among older patients. J. Sport Heal. Sci. 6, 33–43 (2017)
A. Nawaz, N. Skjaeret, J.L. Helbostad, Usability and acceptability of balance exergames in older adults: a scoping review. Health Inf. J. 22(4), 911–931 (2016)
F.W. Simor, M.R. Brum, J.D.E. Schmidt, R. Rieder, A.C.B. de Marchi, Usability evaluation methods for gesture-based games: a systematic review. JMIR Serious Games 4(2), e17 (2016)
D. Kairy, P. Lehoux, C. Vincent, M. Visitin, A systematic review of clinical outcomes, clinical process, healthcare utilization and costs associated with telerehabilitation. Disabil. Rehabil. 31, 427–447 (2009)
J.E. Maddux, Self-efficacy Theory (Springer, New York, 1995)
V. Venkatesh, M.G. Morris, G.B. Davis, F.D. Davis, User acceptance of information technology: toward a unified view. MIS Q. 27(3), 425–478 (2003)
F.D. Davis, Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 13, 319–340 (1989)
Ajzen, The theory of planned behavior. Organ Behav. Hum. Decis. Process 50(2), 179–211 (1991)
R.P. Bagozzi, The legacy of the technology acceptance model and a proposal for a paradigm shift. J. Assoc. Inf. Syst. 8, 244–254 (2007)
J. Nielsen, Usability Engineering (Elsevier, Amsterdam, 1994)
B. Shneiderman, Designing the User Interface: Strategies for Effective Human-Computer Interaction (Pearson Education India, New Delhi, 1986)
K.M. Gerling, J. Schild, M. Masuch, Exergaming for elderly: analyzing player experience and performance. Mensch Comput. (2011)
K.-K. Kang, J.-A. Kim, D. Kim, Development of a sensory gate–ball game system for the aged people. Vis. Comput. 25(12), 1073–1083 (2009)
T.C. Chan, F. Chan, Y.F. Shea, O.Y. Lin, Y.K. Luk, F.H. Chan, Interactive virtual reality Wii in geriatric day hospital: a study to assess its feasibility, acceptability and efficacy. Geriatr. Gerontol. Int. 12(4), 714–721 (2012)
Y.-Y. Chao, Y.K. Scherer, Y.-W. Wu, K.T. Lucke, C.A. Montgomery, The feasibility of an intervention combining self-efficacy theory and Wii Fit exergames in assisted living residents: a pilot study. Geriatr Nurs 34(5), 377–382 (2013)
A.S. Billis, E.I. Konstantinidis, A.I. Ladas, M.N. Tsolaki, C. Pappas, P.D. Bamidis. Evaluating affective usability experiences of an exergaming platform for seniors, in Proceedings of 10th International Workshop on Biomedical Engineering, Kos, October 2011
J.M. Flach, C.O. Dominguez, User-centered design: integrating the user, instrument, and goal. Ergon Des. 19–24 (1995)
K. Isbister, N. Shaffer, Heuristic evaluation of games, in Game Usability: Advancing the Player Experience (CRC Press, 2008), pp. 79–89
R. Bernhaupt, Evaluating user experience in games: concepts and methods, in Human Computer Interaction Series (2010)
P. Sweetser, P. Wyeth, GameFlow: a model for evaluating player enjoyment in games. Comput. Entertain. 3, 3 (2005)
P. Bonato, Advances in wearable technology and applications in physical medicine and rehabilitation. J. Neuroeng. Rehabil. 2, 2 (2005)
S. Obdrzalek, G. Kurillo, F. Ofli, R. Bajcsy E. Seto, H. Jimison, M. Pavel. Accuracy and robustness of kinect pose estimation in the context of coaching of elderly population, in Proceedings of IEEE Engineering in Medicine and Biology Society (EMBC) (2012), pp. 1188–1193
G.C. Burdea, Virtual rehabilitation—benefits and challenges. Methods Inf. Med. 42(519), 1–11 (2003)
M. Kim, C. Jeon, J. Kim. A study on immersion and presence of a portable hand haptic system for immersive virtual reality. Sensors, 17(5), pii: E1141 (2017)
B.G. Witmer, M.J. Singer, Measuring presence in virtual environments: a presence questionnaire. Presence 7, 225–240 (1998)
D. Corbetta, F. Imeri, R. Gatti, Rehabilitation that incorporates virtual reality is more effective than standard rehabilitation for improving walking speed, balance and mobility after stroke: a systematic review. J. Physiother. 61, 117–124 (2015)
N.A. Borghese, M. Pirovano, P.L. Lanzi, S. Wuest, E.D. de Bruin, Computational intelligence and game design for effective at-home stroke rehabilitation. Games Health J. Res. Dev. Clin. Appl. 2(2), 81–88 (2013)
B. Lange, C.Y. Chang, E. Suma, B. Newman, A.S. Rizzo, M. Bolas. Development and evaluation of low cost game-based balance rehabilitation tool using the microsoft kinect sensor, in Proceedings of IEEE Engineering in Medicine and Biology Society, EMBC (2011), pp. 1831–1834
R. Hunicke, M. Le Blanc, R. Zubek, MDA: a formal approach to game design and game research, in Proceedings of AAAI-04 Workshop on Challenges in Game AI, July 2004, pp. 1–5
R. Dillon. The 6-11 framework: a new methodology for game analysis and design, in Proceedings of GAMEON, Singapore, Asia, March 2011, pp. 25–29
V. Stavljanin, M. Minovic, Gamification ecosystems: current state and perspectives, in Open Source Solutions for Knowledge Management and Technological Ecosystems (IGI Global, 2017)
G. Zichermann, C. Cunningham, Gamification by Design. Implementing Game Mechanics in Web and Mobile Apps (O’Reilly Media, Sebastopol, C.A., 2011)
K. Werbach, D. Hunter. For the Win. How Game Thinking Can Revolutionize Your Business. (Wharton Digital Press, 2012)
Dev.Mag. devmag.org.za/2012/04/19/video-game-audio-diegesis-theory-2/
M. Pirovano, E. Surer, R. Mainetti, P.L. Lanzi, N.A. Borghese, Exergaming and rehabilitation: a methodology for the design of effective and safe therapeutic exergames. Entertain. Comput. 14, 55–65 (2016)
Y-L. Theng, A.B. Dahlan, M.L. Akmal, T.Z. Myint. An exploratory study on senior citizens’ perceptions of the Nintento Wii: the case of Singapore, in Proceedings of 3rd International Convention on Rehabilitation Engineering &Assistive Technology, April, Singapore (2009)
B. Bonnechere, B. Jansen, L. Omelina, S. Van Sint, Jan. The use of commercial video games in rehabilitation: a systematic review. Int. J. Rehabil. Res. 39(4), 277–290 (2016)
D. Collado-Mateo, E. Merellano-Navarro, P.R. Olivares, J. Garcia-Rubio, N. Gusi, Effect of exergames on musculoskeletal pain. a systemeatic review and meta-analysis. Scand. J. Med. Sci. Sports 28, 760–771 (2018)
G. Tieri, G. Morone, S. Paolucci, M. Iosa, Virtual reality in cognitive and motor rehabilitation: facts, fiction and fallacies. Expert Rev. Med. Devices 15(2), 107–117 (2018)
R. Aarhus, E. Grönvall, S.B. Larsen, S. Wollsen, Turning training into play: embodied gaming, seniors, physical training and motivation. Gerontechnology 10(2), 110–120 (2011)
M.P.J. Habgood, S.E. Ainsworth, Motivating children to learn effectively: exploring the value of intrinsic integration in educational games. J. Learn. Sci. 20(2), 169–206 (2011)
J. Huberty, L. Ransdell, C. Sidman, Explaining long-term exercise adherence in women who complete a structured exercise program. Res. Q. Exerc. Sport 79, 374 (2008)
S.R. Wood, N. Murillo, P. Bach-y-Rita P, R.S. Leder, J.T. Marks, S.J. Page, Motivating, game-based stroke rehabilitation: a brief report. Top Stroke Rehabil. 10, 134–140 (2003)
Y. Tian, Y. Bian, P. Han, P. Wang, F. Gao, Y. Chen, Physiological signal analysis for evaluating flow during playing of computer games of varying difficulty. Front Psychol. 8, 1121 (2017)
R.M. Ryan, C.S. Rigby, A. Przybylski, The motivational pull of video games: a self-determination theory approach. Motiv. Emot. 30, 347–363 (2006)
R.M. Ryan, E.L. Deci, Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. Am. Psychol. 55, 68–78 (2000)
S. Rigby, R. Richard. Immersion and presence, in Glued to Games: How Video Games Draw Us In and Hold Us Spellbound (2011), pp. 81–96
B.D. Sylvester, M. Standage, J. Dowd, L.J. Martin, S.N. Sweet, M.R. Beauchamp, Perceived variety, psychological needs satisfaction and exercise-related well-being. Psychol. Health 29(9), 1041–1061 (2014)
A. Bandura, Social Foundations of Thought and Action: A Social Cognitive Theory (Prentice-Hall, Englewood Cliffs, N.J., 1986)
M.W. Kreuter, D.W. Farrell, L.R. Olevitch, L.K. Brennan. Tailoring Health Messages: Customizing Communication with Computer Technology (Lawrence Erlbaum Associates, 2000)
D. Thompson, J. Baranowski, K. Cullen, T. Baranowski, Development of a theory-based internet program promoting maintenance of diet and physical activity change to 8-year-old African American girls. Comput. Educ. 48, 446–459 (2007)
L.S. Vygotsky, Mind in Society. The Development of Higher Psychological Processes (1978)
D. Shernoff, M. Csikszentmihalyi, B. Schneider, E. Shernoff, Student engagement in high school classrooms from the perspective of flow theory. Sch. Psychol. Q. 18, 156–176 (2003)
R. Garris, R. Ahlers, J.E. Driskell. Games, motivation, and learning: a research and practice model. Simul. Gaming 441–467 (2002)
B.H. Dobkin, Rehabilitation after stroke. N. Engl. J. Med. 352(16), 1677–1684 (2005)
I-T. Chiang, Old dogs can learn new tricks: exploring effective strategies to facilitate somatosensory video games for institutionalized older veterans, in Edutainment technologies. Educational games and virtual reality/augmented reality applications, ed. by M. Chang, W.-Y. Hwang, M.-P. Chen, et al. (Springer, Berlin)
Acknowledgements
The work was supported by Fundação para a Ciência e a Tecnologia, project PTDC/DTT-DES/6776/2014 and Instituto de Telecomunicações.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Postolache, G. et al. (2019). Serious Games Based on Kinect and Leap Motion Controller for Upper Limbs Physical Rehabilitation. In: Mukhopadhyay, S., Jayasundera, K., Postolache, O. (eds) Modern Sensing Technologies . Smart Sensors, Measurement and Instrumentation, vol 29. Springer, Cham. https://doi.org/10.1007/978-3-319-99540-3_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-99540-3_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-99539-7
Online ISBN: 978-3-319-99540-3
eBook Packages: EngineeringEngineering (R0)