Skip to main content
SpringerLink
Log in

Academic methods for usability evaluation of serious games: a systematic review

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In the last years, there has been an increasing interest in the design of video games as a tool for education, training, health promotion, socialization, etc. Usability, which is a key factor in any video game, becomes even more important in these so-called “serious games”, where the users’ special characteristics should be considered, and the game efficacy depends on the users’ adherence and engagement. However, evaluation of the usability of this kind of games requires a redefinition of techniques, methods and even terminology. In this paper, we elicit six research questions and conduct a systematic review of the scientific literature, which resulted in the selection of 187 papers that contained the most relevant responses. The conclusions of this systematic review illustrate the general status of current academic usability evaluations of these games and the main trends in the selection of methodologies and how are they applied. This view may be a very valuable foundation for future research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Abdulhak SA, Cha KH, Kang DK (2011) Evaluation of online social network games usability using verbal protocol analysis. In: 2011 6th International Conference on Computer Sciences and Convergence Information Technology (ICCIT), pp 7–11. IEEE

  2. Abeele VV, Zaman B, Abeele MV (2008) The unlikeability of a cuddly toy interface: an experimental study of preschoolers’ likeability and usability of a 3D Game played with a cuddly toy versus a keyboard. In Fun and Games, pp 118–131. Springer, Berlin

  3. Adamo-Villani N, Wright K (2007) SMILE: an immersive learning game for deaf and hearing children. In: ACM SIGGRAPH 2007 educators program. ACM, p 17

  4. Ahmad WFW, Shaarani ARS, Afrizal S (2012) Mobile language translation game. In: Computer & Information Science (ICCIS), 2012 International Conference on (Vol. 2), pp 1099–1104. IEEE

  5. Albu M, Atack L, Srivastava I (2015) Simulation and gaming to promote health education: results of a usability test. Health Educ J 74(2):244–254

    Article  Google Scholar 

  6. Ancker JS, Kukafka R (2007) A combined qualitative method for testing an interactive risk communication tool. In: AMIA Annual Symposium Proceedings (Vol. 2007), p 16. American Medical Informatics Association

  7. Ang CS, Avni E, Zaphiris P (2008) Linking pedagogical theory of computer games to their usability. Int J E-Learning 7(3):533–558

    Google Scholar 

  8. Annett J, Duncan KD (1967) Task analysis and training design. Occup Psychol 12:211–221

    Google Scholar 

  9. Arroyo-Palacios J, Romano DM (2009) Exploring the use of a respiratory-computer interface for game interaction. In Games Innovations Conference, 2009. ICE-GIC 2009. International IEEE Consumer Electronics Society’s, pp 154–159. IEEE

  10. Aung YM, Al-Jumaily A, Anam K (2014) A novel upper limb rehabilitation system with self-driven virtual arm illusion. In: 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp 3614–3617. IEEE

  11. Baauw E, Bekker MM, Barendregt W (2005) A structured expert evaluation method for the evaluation of children’s computer games. Proceedings of human–computer interaction INTERACT 2005, Rome, 12–16 September 2005, pp 457–469

  12. Baauw E, Bekker MM, Markopoulos P (2006) Assessing the applicability of the structured expert evaluation method (SEEM) for a wider age group. In: Proceedings of the 2006 conference on Interaction design and children, pp 73–80. ACM

  13. Barr P (2008) Video game values: Play as human-computer interaction. PhD Thesis, Victoria University of Wellington

  14. Bastien J, Scapin D (1991) Ergonomic criteria for the evaluation of human-computer interfaces, Rapport de Recherche N. 156. INRIA, France

    Google Scholar 

  15. Bekker MM, Baauw E, Barendregt W (2008) A comparison of two analytical evaluation methods for educational computer games for young children. Cogn Tech Work 10(2):129–140

    Article  Google Scholar 

  16. Bellotti F, Berta R, De Gloria A, D’ursi A, Fiore V (2012) A serious game model for cultural heritage. J Comput Cult Herit 5(4):17

    Article  Google Scholar 

  17. Bellotti F, Ferretti E, De Gloria A (2005) Discovering the European heritage through the ChiKho educational web game. In: Intelligent Technologies for Interactive Entertainment, pp 13–22. Springer, Berlin

  18. Benveniste S, Jouvelot P, Péquignot R (2010) The MINWii Project: Renarcissization of patients suffering from Alzheimer’s Disease through video game-based music therapy. In: Entertainment Computing-ICEC 2010, pp 79–90. Springer, Berlin

  19. Bernhaupt R, Schwaiger D, Riegler S, Enthaler D (2007) Evaluating children’s gaming experiences. In: Proceedings of the international conference on Advances in computer entertainment technology, pp 224–225. ACM

  20. Bernsen NO, Dybkjær H, Dybkjær L (1994) Wizard of oz prototyping: how and when. Proc. CCI Working Papers Cognit. Sci./HCI, Roskilde, Denmark

  21. Bias R (1991) Interface-walkthroughs: efficient collaborative testing. IEEE Softw 8(5):94–95

    Article  Google Scholar 

  22. Bias RG (1994) The pluralistic usability walkthrough: coordinated empathies. In: Usability inspection methods. John Wiley & Sons, Inc., p 63–76

  23. Boulay M, Benveniste S, Boespflug S, Jouvelot P, Rigaud AS (2011) A pilot usability study of MINWii, a music therapy game for demented patients. Technol Health Care 19(4):233–246

    Google Scholar 

  24. Brereton P, Kitchenham BA, Budgen D, Turner M, Khalil M (2007) Lessons from applying the systematic literature review process within the software engineering domain. J Syst Softw 80(4):571–583

    Article  Google Scholar 

  25. Bronner S, Pinsker R, Noah JA (2015) Physiological and psychophysiological responses in experienced players while playing different dance exer-games. Comput Hum Behav 51:34–41

    Article  Google Scholar 

  26. Brooke J (1996) SUS-A quick and dirty usability scale. Usability Eval Ind 189(194):4–7

    Google Scholar 

  27. Brown DJ, McHugh D, Standen P, Evett L, Shopland N, Battersby S (2011) Designing location-based learning experiences for people with intellectual disabilities and additional sensory impairments. Comput Educ 56(1):11–20

    Article  Google Scholar 

  28. Browne K, Anand C (2013) Gamification and serious game approaches for introductory computer science tablet software. In: Proceedings of the First International Conference on Gameful Design, Research, and Applications, pp 50–57. ACM

  29. Brown-Johnson CG, Berrean B, Cataldo JK (2015) Development and usability evaluation of the mHealth Tool for Lung Cancer (mHealth TLC): a virtual world health game for lung cancer patients. Patient Educ Couns 98(4):506–511

    Article  Google Scholar 

  30. Burns M, Colbert M (2013) Web analytics: the new purpose towards predictive mobile games. In: Advances in computer entertainment, pp 1–13. Springer International Publishing

  31. Cabral C, Dehanov J, Dias JMS, Bastos R (2005) Developing games with Magic Playground: a gesture-based game engine. In: Proceedings of the 2005 ACM SIGCHI International Conference on Advances in computer entertainment technology, pp 361–362. ACM

  32. Cameiräo MS, Badia SB, Oller ED, Verschure PFMJ (2010) Neurorehabilitation using the virtual reality based Rehabilitation Gaming System: methodology, design, psychometrics, usability and validation. J Neuroeng Rehabil 7(1):48

    Article  Google Scholar 

  33. Cao Y, McGill WL (2013) LinkIT: a Ludic elicitation game for eliciting risk perceptions. Risk Anal 33(6):1066–1082

    Article  Google Scholar 

  34. Caputo FM, Giachetti A (2015) Evaluation of basic object manipulation modes for low-cost immersive Virtual Reality. In: Proceedings of the 11th Biannual Conference on Italian SIGCHI Chapter, pp 74–77. ACM

  35. Carrigy T, Naliuka K, Paterson N, Haahr M (2010) Design and evaluation of player experience of a location-based mobile game. In: Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries, pp 92–101. ACM

  36. Carter C, Mehdi Q, Hartley T (2012) Navigational techniques to improve usability and user experience in RPG games. In: 2012 17th International Conference on Computer Games (CGAMES), pp 86–93. IEEE

  37. Centieiro P, Romão T, Dias AE (2011) A location-based multiplayer mobile game to encourage pro-environmental behaviours. In: Proceedings of the 8th International Conference on Advances in Computer Entertainment Technology, pp 31. ACM

  38. Chammas A, Moraes A, Teixeira EA (2011) Ergonomics and usability of children interfaces: spore®, the case study. Work (Reading, Mass) 41:1024–1031

    Google Scholar 

  39. Chang SB, Deng YC, Cheng HN, Liao HC, Yu FY, Chan TW (2007) Implementation and evaluation of EduBingo for arithmetic drill. In: The First IEEE International Workshop on Digital Game and Intelligent Toy Enhanced Learning. DIGITEL’07. pp 99–103. IEEE

  40. Chee YS, Tan EM, Liu Q (2010) Statecraft X: enacting citizenship education using a mobile learning game played on Apple iPhones. In: 2010 6th IEEE International Conference on Wireless, Mobile and Ubiquitous Technologies in Education (WMUTE), pp 222–224. IEEE

  41. Chen KH, Shen KS, Ma MY (2012) The functional and usable appeal of Facebook SNS games. Internet Res 22(4):467–481

    Article  Google Scholar 

  42. Choi YJ (2009) Providing novel and useful data for game development using usability expert evaluation and testing. In: Sixth International Conference on Computer Graphics, Imaging and Visualization, 2009. CGIV’09. pp 129–132. IEEE

  43. Chung KM, Shin DH (2015) Effect of elastic touchscreen and input devices with different softness on user task performance and subjective satisfaction. Int J Hum Comput Stud 83:12–26

    Article  Google Scholar 

  44. Clanton C (1998) An interpreted demonstration of computer game design. In: CHI 98 Conference Summary on Human Factors in Computing Systems. ACM, p 1–2

  45. Clawson J, Patel N, Starner T (2010) Dancing in the streets: the design and evaluation of a wearable health game. In: 2010 International Symposium on Wearable Computers (ISWC), pp 1–4. IEEE

  46. Cornett S (2004) The usability of massively multiplayer online roleplaying games: designing for new users. In: Proceedings of the SIGCHI conference on Human factors in computing systems, pp 703–710. ACM

  47. Couceiro RM, Papastergiou M, Kordaki M, Veloso AI (2013) Design and evaluation of a computer game for the learning of Information and Communication Technologies (ICT) concepts by physical education and sport science students. Educ Inf Technol 18(3):531–554

    Article  Google Scholar 

  48. Cruz VT, Pais J, Bento V, Mateus C, Colunas, M., Alves I, … Rocha NP (2013) A rehabilitation tool designed for intensive web-based cognitive training: description and usability study. JMIR Res Protocols, 2(2)

  49. Cybis W, Holtz A, Faust R (2010) Ergonomia e usabilidade. Novatec, São Paulo

    Google Scholar 

  50. Dahlan Bin Abdul Ghani (2015) Wayang fighters: preserving the art of Wayang Kulit (shadow puppets) through game application. In: Proceedings of the 9th International Conference on Ubiquitous Information Management and Communication (IMCOM ‘15). ACM, New York, Article 26:7

  51. de Carvalho BJA, Soares MM, das Neves AMM, Medeiros RP (2013) Interactive doodles: a comparative analysis of the usability and playability of google trademark games between 2010 and 2012. In: Design, user experience, and usability. Health, learning, playing, cultural, and cross-cultural user experience, pp 508–517. Springer, Berlin

  52. DeShazo J, Harris L, Turner A, Pratt W (2010) Designing and remotely testing mobile diabetes video games. J Telemed Telecare 16(7):378–382

    Article  Google Scholar 

  53. Desurvire H, Caplan M, Toth JA (2004) Using heuristics to evaluate the playability of games. In: CHI‘04 extended abstracts on Human factors in computing systems. ACM, p 1509–1512

  54. Deterding S, Dixon D, Khaled R, Nacke L (2011) From game design elements to gamefulness: defining gamification. In: Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments, pp 9–15. ACM

  55. Diah NM, Ismail M, Ahmad S, Dahari MKM (2010) Usability testing for educational computer game using observation method. In: 2010 International Conference on Information Retrieval & Knowledge Management, (CAMP), pp 157–161. IEEE

  56. Diehl LA, Souza RM, Alves JB, Gordan PA, Esteves RZ, Jorge MLSG, Coelho ICM (2013) InsuOnline, a serious game to teach insulin therapy to primary care physicians: design of the game and a randomized controlled trial for educational validation. JMIR Res Protocols, 2(1)

  57. Direito A, Jiang Y, Whittaker R, Maddison R (2015) Smartphone apps to improve fitness and increase physical activity among young people: protocol of the Apps for IMproving FITness (AIMFIT) randomized controlled trial. BMC Public Health 15(1):635

    Article  Google Scholar 

  58. Dores AR, Carvalho IP, Barbosa F, Almeida I, Guerreiro S, Leitäo M, … Castro-Caldas A (2011) Serious games: are they part of the solution in the domain of cognitive rehabilitation?. In: Serious Games Development and Applications, pp 95–105. Springer, Berlin

  59. Dyck J, Pinelle D, Brown B, Gutwin C (2003) Learning from games: HCI design innovations in entertainment software. In: Graphics interface, p 237–246

  60. Ebner M, Holzinger A (2007) Successful implementation of user-centered game based learning in higher education: an example from civil engineering. Comput Educ 49(3):873–890

    Article  Google Scholar 

  61. Eladhari MP, Ollila EM (2012) Design for research results experimental prototyping and play testing. Simul Gaming 43(3):391–412

    Article  Google Scholar 

  62. El-Tayeh A, Gil N, Freeman J (2008) A methodology to evaluate the usability of digital socialization in ‘virtual’ engineering design. Res Eng Des 19(1):29–45

    Article  Google Scholar 

  63. Fabricatore C, Nussbaum M, Rosas R (2002) Playability in action videogames: a qualitative design model. Hum Comput Interact 17(4):311–368

    Article  Google Scholar 

  64. FDIS, I. 9241-210 (2009) Ergonomics of human system interaction-Part 210: human-centered design for interactive systems

  65. Federoff MA (2002) Heuristics and usability guidelines for the creation and evaluation of fun in video games (Doctoral dissertation, Indiana University)

  66. Ferracani A, Pezzatini D, Seidenari L, Del Bimbo A (2014) Natural and virtual environments for the training of emergency medicine personnel. Univ Access Inf Soc, 1–12

  67. Ferraz M, Romão T, Câmara A (2010) The Alberto’s Gravimente Toys: children’s fiction on technological design. In: Proceedings of the 7th International Conference on Advances in Computer Entertainment Technology, pp 7–10. ACM

  68. Ferrer V, Perdomo A, Rashed-Ali H, Fies C, Quarles J (2013) How does usability impact motivation in augmented reality serious games for education?. In: 2013 5th International Conference on Games and virtual worlds for serious applications (VS-GAMES), pp 1–8. IEEE

  69. Fitzgerald D, Trakarnratanakul N, Dunne L, Smyth B, Caulfield B (2008) Development and user evaluation of a virtual rehabilitation system for wobble board balance training. In: Engineering in Medicine and Biology Society, 2008. EMBS 2008. 30th Annual International Conference of the IEEE, pp 4194–4198. IEEE

  70. Fitz-Walter Z, Tjondronegoro D, Koh D, Zrobok M (2012) Mystery at the library: encouraging library exploration using a pervasive mobile game. In: Proceedings of the 24th Australian Computer-Human Interaction Conference, pp 142–145. ACM

  71. Fitz-Walter Z, Wyeth P, Tjondronegoro D, Scott-Parker B (2013) Driven to drive: Designing gamification for a learner logbook smartphone application. In: Proceedings of the First International Conference on Gameful Design, Research, and Applications, pp 42–49. ACM

  72. Franck LS, Jones M (2003) Computer-taught coping techniques for venepuncture: preliminary findings from usability testing with children, parents and staff. J Child Health Care 7(1):41–54

    Article  Google Scholar 

  73. Frutos-Pascual M, Zapirain BG, Zorrilla AM (2014) Adaptive tele-therapies based on serious games for health for people with time-management and organisational problems: preliminary results. Int J Environ Res Public Health 11(1):749–772

    Article  Google Scholar 

  74. Fullerton T, Swain C, Hoffman S (2004) Game design workshop: designing, prototyping, & playtesting games. CRC Press

  75. Fulton B, Romero R (2004) User-testing in a hostile environment: overcoming resistance and Apathy in your game company. In: Game Developer’s Conference, San Jose CA

  76. Furió D, González-Gancedo S, Juan MC, Seguí I, Rando N (2013) Evaluation of learning outcomes using an educational iPhone game vs. traditional game. Comput Educ 64:1–23

    Article  Google Scholar 

  77. Furió D, Juan MC, Seguí I, Vivó R (2015) Mobile learning vs. traditional classroom lessons: a comparative study. J Comput Assist Learn 31(3):189–201

    Article  Google Scholar 

  78. Gamberini L, Corradi N, Zamboni L, Perotti M, Cadenazzi C, Mandressi S, … Aman P (2011) Saving is fun: designing a persuasive game for power conservation. In: Proceedings of the 8th International Conference on Advances in Computer Entertainment Technology, p 16. ACM

  79. Ganesan S, Anthony L (2012) Using the kinect to encourage older adults to exercise: a prototype. In: CHI‘12 Extended Abstracts on Human Factors in Computing Systems, pp 2297–2302. ACM

  80. Gerling KM, Klauser M, Niesenhaus J (2011) Measuring the impact of game controllers on player experience in FPS games. In: Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments, pp 83–86. ACM

  81. Gilleade KM, Dix A (2004) Using frustration in the design of adaptive videogames. In: Proceedings of the 2004 ACM SIGCHI International Conference on Advances in computer entertainment technology. ACM, p 228–232

  82. Goh DHL, Lee CS, Chua AYK (2010) Do games motivate mobile content sharing? Springer, Berlin, pp 61–70

    Google Scholar 

  83. Goh DHL, Razikin K, Chua AY, Lee CS, Tan KT (2011) Understanding location-based information sharing in a mobile human computation game. In: Internet of things (iThings/CPSCom), 2011 International Conference on and 4th International Conference on Cyber, Physical and Social Computing, pp 209–216. IEEE

  84. Gonçalves D, Jesus R, Grangeiro F, Romão T, Correia N (2008) Tag around: a 3D gesture game for image annotation. In: Proceedings of the 2008 International Conference on Advances in Computer Entertainment Technology, pp 259–262. ACM

  85. González C, Muñoz V, Toledo P, Mora A, Moreno L (2014) EMATIC: an inclusive educational application for tablets. In: Proceedings of the XV International Conference on Human Computer Interaction, p. 94. ACM

  86. Goodman D, Bradley NL, Paras B, Williamson IJ, Bizzochi J (2006) Video gaming promotes concussion knowledge acquisition in youth hockey players. J Adolesc 29(3):351–360

    Article  Google Scholar 

  87. Guidali M, Duschau-Wicke A, Broggi S, Klamroth-Marganska V, Nef T, Riener R (2011) A robotic system to train activities of daily living in a virtual environment. Med Biol Eng Comput 49(10):1213–1223

    Article  Google Scholar 

  88. Gürkök H, Hakvoort G, Poel M, Nijholt A (2011) User expectations and experiences of a speech and thought controlled computer game. In: Proceedings of the 8th International Conference on Advances in Computer Entertainment Technology, p 53. ACM

  89. Gürkök H, Hakvoort G, Poel M (2011) Evaluating user experience in a selection based brain-computer interface game A comparative study. In: Entertainment computing-ICEC 2011, pp 77–88. Springer, Berlin

  90. Ha J, Park S (2013) A comparative study by age in evaluating web-based 3D model house. Archit Res 15(4):175–182

    Article  Google Scholar 

  91. Haggerty K, Yadamsuren B, Kibaru F, Basic J (2013) Children’s information needs of a patient engagement system in a hospital room: insights from usability testing in a simulation laboratory. Proc Am Soc Inf Sci Technol 50(1):1–5

    Article  Google Scholar 

  92. Halic T, Kockara S, Demirel D, Willey M, Eichelberger K (2014) MoMiReS: Mobile mixed reality system for physical & occupational therapies for hand and wrist ailments. In: Innovations in Technology Conference (InnoTek), 2014 IEEE, pp 1–6. IEEE

  93. Hannig A, Kuth N, Özman M, Jonas S, Spreckelsen C (2012) eMedOffice: a web-based collaborative serious game for teaching optimal design of a medical practice. BMC Med Educ 12(1):104

    Article  Google Scholar 

  94. Hart P, Bierwirth R, Fulk G, Sazonov E (2014) The design and evaluation of an activity monitoring user interface for people with stroke. In Engineering in Medicine and Biology Society (EMBC), 2014 36th Annual International Conference of the IEEE, pp 5908–5911. IEEE

  95. Hart SG, Staveland LE (1988) Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. Adv Psychol 52:139–183

    Article  Google Scholar 

  96. Heo J, Ham DH, Park S, Song C, Yoon WC (2009) A framework for evaluating the usability of mobile phones based on multi-level, hierarchical model of usability factors. Interact Comput 21(4):263–275

    Article  Google Scholar 

  97. Hertzum M, Jacobsen NE (2001) The evaluator effect: a chilling fact about usability evaluation methods. Int J Hum Comput Interact 13(4):421–443

    Article  Google Scholar 

  98. Hou HT, Li MC (2014) Evaluating multiple aspects of a digital educational problem-solving-based adventure game. Comput Hum Behav 30:29–38

    Article  Google Scholar 

  99. Höysniemi J, Aula A, Auvinen P, Hännikäinen J, Hämäläinen P (2004) Shadow boxer: a physically interactive fitness game. In: Proceedings of the third Nordic conference on Human-computer interaction, pp 389–392. ACM

  100. Hwang MY, Hong JC, Jong JT, Lee CK, Chang HY (2009) From fingers to embodiment: A study on the relations of the usability, dependability of the embodied interactive video games and the elders‚ Äô flow experience. In: Learning by Playing. Game-based Education System Design and Development, pp 464–472. Springer, Berlin

  101. Hwang W, Salvendy G (2010) Number of people required for usability evaluation: the 10¬±2 rule. Commun ACM 53(5):130–133

    Article  Google Scholar 

  102. Infante C, Weitz J, Reyes T, Nussbaum M, Gómez F, Radovic D (2010) Co-located collaborative learning video game with single display groupware. Interact Learn Environ 18(2):177–195

    Article  Google Scholar 

  103. Isleyen F, Gulkesen KH, Cinemre B, Samur MK, Zayim N, Sen KS (2013) Evaluation of the usability of a serious game aiming to teach facial expressions to schizophrenic patients. Stud Health Technol Inform 205:662–666

    Google Scholar 

  104. Ismail M, Diah NM, Ahmad S, Kamal NAM, Dahari MKM (2011) Measuring usability of educational computer games based on the user success rate. In: 2011 International Symposium on Humanities, Science & Engineering Research (SHUSER), pp 56–60. IEEE

  105. ISO, S. 9241-11 (1998) Ergonomic requirements for office work with visual display terminals (VDTs)-Part II guidance on usability

  106. Istance H, Hyrskykari A, Vickers S, Chaves T (2009) For your eyes only: controlling 3d online games by eye-gaze. In: Human-Computer Interaction-INTERACT 2009, pp 314–327. Springer, Berlin

  107. Ivory MY, Hearst MA (2001) The state of the art in automating usability evaluation of user interfaces. ACM Comput Surv 33(4):470–516

    Article  Google Scholar 

  108. Jett J, Sacchi S, Lee JH, Clarke RI (2016) A conceptual model for video games and interactive media. J Assoc Inf Sci Technol 67(3):505–517

    Article  Google Scholar 

  109. Jimenez-Diaz G, Gonzalez-Calero PA, Gomez-Albarran M (2012) Role-play virtual worlds for teaching object-oriented design: the ViRPlay development experience. Softw: Pract Exp 42(2):235–253

    Google Scholar 

  110. Jørgensen AH (2004) Marrying HCI/Usability and computer games: a preliminary look. In: Proceedings of the third Nordic conference on Human-computer interaction. ACM, p 393–396

  111. Ju E, Wagner C (1997) Personal computer adventure games: their structure, principles, and applicability for training. SIGMIS Database 28(2):78–92

    Article  Google Scholar 

  112. Karat J (1997) User-centered software evaluation methodologies. Handb Hum Comput Interact 2:689–704

    Article  Google Scholar 

  113. Karouzaki E, Savidis A (2012) A framework for adaptive game presenters with emotions and social comments. Int J Comput Games Technol 2012:6

    Article  Google Scholar 

  114. Khan DU, Ananthanarayan S, Le AT, Schaefbauer CL, Siek K (2012) Designing mobile snack application for low socioeconomic status families. In: 2012 6th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth), pp 57–64. IEEE

  115. Kim YS, Collins M, Bulmer W, Sharma S, Mayrose J (2013) Haptics Assisted Training (HAT) System for children’s handwriting. In World Haptics Conference (WHC), 2013, pp 559–564. IEEE

  116. Kim SL, Suk HJ, Kang JH, Jung JM, Laine TH, Westlin J (2014) Using unity 3D to facilitate mobile augmented reality game development. In: 2014 I.E. World Forum on Internet of Things (WF-IoT), pp 21–26. IEEE

  117. Kistler F, André E (2015) How can I Interact?: comparing full body gesture visualizations. In: Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play, pp 583–588. ACM

  118. Kitchenham B, Brereton P (2013) A systematic review of systematic review process research in software engineering. Inf Softw Technol 55(12):2049–2075

    Article  Google Scholar 

  119. Klisch Y, Miller LM, Beier ME, Wang S (2012) Teaching the biological consequences of alcohol abuse through an online game: impacts among secondary students. CBE Life Sci Educ 11(1):94–102

    Article  Google Scholar 

  120. Klisch Y, Miller LM, Wang S, Epstein J (2012) The impact of a science education game on students’ learning and perception of inhalants as body pollutants. J Sci Educ Technol 21(2):295–303

    Article  Google Scholar 

  121. Konstantinidis EI, Billis AS, Mouzakidis CA, Zilidou VI, Antoniou PE, Bamidis PD (2016). Design, implementation, and wide pilot deployment of fit for all: an easy to use exergaming platform improving physical fitness and life quality of senior citizens. IEEE J Biomed Health Inform. 20(1):189–200

  122. Korhonen H (2010) Comparison of playtesting and expert review methods in mobile game evaluation. In: Proceedings of the 3rd International Conference on Fun and Games. ACM, p 18–27

  123. Korhonen H, Koivisto EM (2006) Playability heuristics for mobile games. In: Proceedings of the 8th conference on Human-computer interaction with mobile devices and services. ACM, p 9–16

  124. Kos’myna N, Tarpin-Bernard F (2013) Evaluation and Comparison of a Multimodal Combination of BCI Paradigms with Consumer-Grade Hardware and Eye Tracking. IEEE Transactions on Computational Intelligence and AI in Games (in print, 2013)

  125. Koster R (2013) Theory of fun for game design. O’Reilly Media, Inc

  126. Kothandapani S, Goh DHL, Razikin K (2012) Influence of playability and usability in a mobile human computation game. In: 2012 32nd International Conference on Distributed Computing Systems Workshops (ICDCSW), pp 75–78. IEEE

  127. Krebs P, Burkhalter JE, Snow B, Fiske J, Ostroff JS (2013) Development and alpha testing of QuitIT: an interactive video game to enhance skills for coping with smoking urges. JMIR Res Protocols, 2(2)

  128. Laine TH, Sedano CAI, Joy M, Sutinen E (2010) Critical factors for technology integration in game-based pervasive learning spaces. IEEE Trans Learn Technol 3(4):294–306

    Article  Google Scholar 

  129. Laitinen S (2006) Do usability expert evaluation and test provide novel and useful data for game development? J Usability Stud 2(1):64–75

    Google Scholar 

  130. Lange B, Flynn S, Proffitt R, Chang CY, Rizzo AS (2010) Development of an interactive game-based rehabilitation tool for dynamic balance training. Top Stroke Rehabil 17(5):345–352

    Article  Google Scholar 

  131. Lee TS, Goh SJA, Quek SY, Phillips R, Guan C, Cheung YB, … Krishnan KRR (2013) A brain-computer interface based cognitive training system for healthy elderly: a randomized control pilot study for usability and preliminary efficacy

  132. Lee PW, Wang HY, Tung YC, Lin JW, Valstar A (2015) TranSection: hand-based interaction for playing a game within a virtual reality game. In: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, pp 73–76. ACM

  133. Lee PW, Wang HY, Tung YC, Lin JW, Valstar A (2015) TranSection: hand-based interaction for playing a game within a virtual reality game. In: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA ‘15). ACM, New York, p 73–76

  134. Lewis JR (1995) IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Hum Comput Inter 7(1):57–78

    Article  Google Scholar 

  135. Lewis C, Polson PG, Wharton C, Rieman J (1990) Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces. In: Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, p 235–242

  136. Liao LD, Chen CY, Wang IJ, Chen SF, Li SY, Chen BW, … Lin CT (2012) Gaming control using a wearable and wireless EEG-based brain-computer interface device with novel dry foam-based sensors. J Neuroeng Rehabil 9(1): 5

  137. Lin HCK, Hsieh MC, Wang CH, Sie ZY, Chang SH (2011) Establishment and usability evaluation of an interactive AR learning system on conservation of fish. Turk Online J Educ Technol 10(4):181–187

    Google Scholar 

  138. Liu B, Ji Y, Zhang C (2010) A Study and Realization of Mobile Social Network System. In: 2010 Proceedings of the 5th International Conference on Ubiquitous Information Technologies and Applications (CUTE), pp 1–4. IEEE

  139. Livingston IJ, Mandryk RL, Stanley KG (2010) Critic-proofing: how using critic reviews and game genres can refine heuristic evaluations. In: Proceedings of the International Academic Conference on the Future of Game Design and Technology. ACM, p 48–55

  140. Llorens R, Alcañiz M, Navarro MD, Ferri J, Noe E (2013) Self-awareness rehabilitation through a multi-touch virtual game board after acquired brain injury. In: 2013 International Conference on Virtual Rehabilitation (ICVR), pp 134–138. IEEE

  141. Lu Z, Lal Khan MS, Ur Réhman S (2013) Hand and foot gesture interaction for handheld devices. In: Proceedings of the 21st ACM International conference on Multimedia, pp 621–624. ACM

  142. Lugrin JL, Cavazza M, Charles F, Le Renard M, Freeman J, Lessiter J (2013) Immersive fps games: user experience and performance. In: Proceedings of the 2013 ACM international workshop on Immersive media experiences, pp 7–12. ACM

  143. Luojus P, Koskela J, Ollila K, Mäki SM, Kulpa-Bogossia R, Heikkinen T, Ojala T (2013) Wordster: collaborative versus competitive gaming using interactive public displays and mobile phones. In: Proceedings of the 2nd ACM International Symposium on Pervasive Displays, pp 109–114. ACM

  144. Lv Z, Halawani A, Fen S, Li H (2015) Touch-less interactive augmented reality game on vision based wearable device. arXiv preprint arXiv:1504.06359

  145. Malone TW (1982) Heuristics for designing enjoyable user interfaces: lessons from computer games. In: Proceedings of the 1982 conference on Human factors in computing systems. ACM, p 63–68

  146. Marco J, Baldassarri S, Cerezo E (2013) NIKVision: developing a tangible application for and with children. J UCS 19(15):2266–2291

    Google Scholar 

  147. Marco J, Cerezo E, Baldassarri S (2013) Bringing tabletop technology to all: evaluating a tangible farm game with kindergarten and special needs children. Pers Ubiquit Comput 17(8):1577–1591

    Article  Google Scholar 

  148. Martins T, Romão T, Sommerer C, Mignonneau L, Correia N (2008) Towards an interface for untethered ubiquitous gaming. In: Proceedings of the 2008 International Conference on Advances in Computer Entertainment Technology, pp 26–33. ACM

  149. Miller LM, Chang CI, Wang S, Beier ME, Klisch Y (2011) Learning and motivational impacts of a multimedia science game. Comput Educ 57(1):1425–1433

    Article  Google Scholar 

  150. Moreno-Ger P, Torrente J, Hsieh YG, Lester WT (2012) Usability testing for serious games: making informed design decisions with user data. Adv Hum Comput Interact 2012:4

    Article  Google Scholar 

  151. Morón MJ, Yáñez R, Cascado D, Suárez-Mejías C, Sevillano JL (2014) A mobile memory game for patients with Acquired Brain Damage: a preliminary usability study. In: 2014 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI), pp 302–305. IEEE

  152. Moustakas K, Tzovaras D, Dybkjær L, Bernsen NO (2009) A modality replacement framework for the communication between blind and hearing impaired people. In: Universal access in human-computer interaction. Applications and services, pp 226–235. Springer, Berlin

  153. Nam CS, Johnson S, Li Y, Seong Y (2009) Evaluation of human-agent user interfaces in multi-agent systems. Int J Ind Ergon 39(1):192–201

    Article  Google Scholar 

  154. Nielsen J (1994) Heuristic evaluation. Usability Inspection Methods 17(1):25–62

    Google Scholar 

  155. Nielsen J (1994) Usability inspection methods. In Conference companion on Human factors in computing systems, pp 413–414. ACM

  156. Nielsen J (1997) The use and misuse of focus groups. IEEE Softw 14(1):94–95

    Article  MathSciNet  Google Scholar 

  157. Nielsen J (2000) Why you only need to test with 5 users. Current Issues in Web Usability, Alertbox

  158. Nielsen J, Clemmensen T, Yssing C (2002) Getting access to what goes on in people’s heads?: reflections on the think-aloud technique. In: Proceedings of the second Nordic conference on Human-computer interaction, pp 101–110. ACM

  159. Nijenhuis SM, Prange GB, Amirabdollahian F, Sale P, Infarinato F, Nasr N, … Rietman JS (2015) Feasibility study into self-administered training at home using an arm and hand device with motivational gaming environment in chronic stroke. J Neuroeng Rehabil 12(1): 1

  160. Novick D, Vicario J, Santaella B, Gris I (2014) Empirical analysis of playability vs. usability in a computer game. In: Design, user experience, and usability. User experience design for diverse interaction platforms and environments. Springer International Publishing, p 720–731

  161. Ntoa S, Adami I, Prokopiou G, Antona M, Stephanidis C (2011) A-cross: an accessible crossword puzzle for visually impaired users. In: Universal Access in Human-Computer Interaction. Users Diversity, pp 342–351. Springer, Berlin

  162. Nystrom D, Nalbandian A, Reichenbach M, O’Keefe B (2012) From exercise to socialize: Improving peer relationships in the fitness center. In: 2012 I.E. International Games Innovation Conference (IGIC), pp 1–5. IEEE

  163. Ollila E (2009) Using prototyping and evaluation methods in iterative design of innovative mobile games. Unpublished doctoral dissertation). Tampere University of Technology, Finland

  164. Olsen T, Procci K, Bowers C (2011) Serious games usability testing: how to ensure proper usability, playability, and effectiveness. In: Design, user experience, and usability. Theory, methods, tools and practice. Springer, Berlin, p 625–634

  165. Paavilainen J, Korhonen HJ, Alha K (2014) Common playability problems in social network games. In: CHI‘14 Extended Abstracts on Human Factors in Computing Systems, pp 1519–1524. ACM

  166. Pagulayan RJ, Keeker K, Wixon D, Romero RL, Fuller T (2002) User-centered design in games. In: Jacko JA, Sears A (eds) The human-computer interaction handbook. L. Erlbaum Associates Inc., Hillsdale, pp 883–906

    Google Scholar 

  167. Paliokas I, Arapidis C, Mpimpitsos M (2011) Playlogo 3d: A 3d interactive video game for early programming education: Let logo be a game. In: 2011 Third International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES), pp 24–31. IEEE

  168. Papaloukas S, Stoli C, Patriarcheas K, Xenos M (2010) A survey on how online games usability affects the retention of information. In: 2010 14th Panhellenic Conference on Informatics (PCI), pp 209–213. IEEE

  169. Pastor I, Hayes HA, Bamberg SJ (2012) A feasibility study of an upper limb rehabilitation system using kinect and computer games. In: 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp 1286–1289. IEEE

  170. Pierotti D (1995) Heuristic evaluation-a system checklist. Xerox Corporation

  171. Pinelle D, Wong N, Stach T, Gutwin C (2009) Usability heuristics for networked multiplayer games. In: Proceedings of the ACM 2009 international conference on Supporting group work. ACM, p 169–178

  172. Pinelle D, Wong N, Stach T (2008) Heuristic evaluation for games: usability principles for video game design. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, p 1453–1462

  173. Plow M, Finlayson M (2014) A qualitative study exploring the usability of Nintendo Wii Fit among persons with multiple sclerosis. Occup Ther Int 21(1):21–32

    Article  Google Scholar 

  174. Pol MH, Lagro J, Fluit LR, Lagro, ÄêJanssen TL, Rikkert O, Marcel GM (2014) Teaching geriatrics using an innovative, individual-centered educational game: students and educators win. A proof-of-concept study. J Am Geriatr Soc 62(10):1943–1949

    Article  Google Scholar 

  175. Polson PG, Lewis C, Rieman J, Wharton C (1992) Cognitive walkthroughs: a method for theory-based evaluation of user interfaces. Int J Man Mach Stud 36(5):741–773

    Article  Google Scholar 

  176. Pranantha D, Bellotti F, Berta R, De Gloria A (2012) A format of serious games for higher technology education topics: a case study in a digital electronic system course. In: 2012 I.E. 12th International Conference on Advanced Learning Technologies (ICALT), pp 13–17. IEEE

  177. Preece J, Sharp H, Rogers Y (2015) Interaction design-beyond human-computer interaction. Wiley

  178. Proctor MD, Panko M, Donovan SJ (2004) Considerations for training team situation awareness and task performance through PC-gamer simulated multiship helicopter operations. Int J Aviat Psychol 14(2):191–205

    Article  Google Scholar 

  179. Raisamo R, Hippula A, Tuominen E, Pasto V, Hasu M (2006) Testing usability of multimodal applications with visually impaired children. IEEE MultiMedia 3:70–76

    Article  Google Scholar 

  180. Rand D, Kizony R, Weiss PTL (2008) The Sony PlayStation II EyeToy: low-cost virtual reality for use in rehabilitation. J Neurol Phys Ther 32(4):155–163

    Article  Google Scholar 

  181. Raybourn EM, Bos N (2005) Design and evaluation challenges of serious games. In: CHI‘05 Extended Abstracts on Human Factors in Computing Systems. ACM, p 2049–2050

  182. Rebolledo-Mendez G, Dunwell I, Martínez-Mirón EA, Vargas-Cerdán MD, De Freitas S, Liarokapis F, García-Gaona AR (2009) Assessing Neurosky’s usability to detect attention levels in an assessment exercise. In: Human-computer interaction. New Trends, pp 149–158. Springer, Berlin

  183. Regenbrecht H, Hoermann S, McGregor G, Dixon B, Franz E, Ott C, Hale L, Schubert T, Hoermann J (2012) Visual manipulations for motor rehabilitation. Comput Graph 36(7):819–834, ISSN 0097–8493

    Article  Google Scholar 

  184. Regenbrecht H, McGregor G, Ott C, Hoermann S, Schubert T, Hale L, … Franz E (2011) Out of reach?-A novel AR interface approach for motor rehabilitation. In: 2011 10th IEEE International Symposium on Mixed and augmented reality (ISMAR), pp 219–228. IEEE

  185. Rehm M, Leichtenstern K (2012) Gesture-based mobile training of intercultural behavior. Multimedia Systems 18(1):33–51

    Article  Google Scholar 

  186. Reichlin L, Mani N, McArthur K, Harris AM, Rajan N, Dacso CC (2011) Assessing the acceptability and usability of an interactive serious game in aiding treatment decisions for patients with localized prostate cancer. J Med Internet Res 13(1)

  187. Reβin M, Haffner C (2007) Human-centered design of interactive TV games with SMS backchannel. In: Interactive TV: a shared experience, pp 37–46. Springer, Berlin

  188. Rias RM, Abdullah M, Salam NHA (2011) A multimedia learning application for undergraduates in Computer Science: The design, development and usability. In: 2011 3rd International Congress on Engineering Education (ICEED), pp 169–175. IEEE

  189. Rice M, Wan M, Foo MH, Ng J, Wai Z, Kwok J, … Teo L (2011) Evaluating gesture-based games with older adults on a large screen display. In: Proceedings of the 2011 ACM SIGGRAPH Symposium on Video Games, pp 17–24. ACM

  190. Rogers K, Frommel J, Breier L, Celik S, Kramer H, Kreidel S, Brich J, Riemer V, Schrader C (2015) Mobile augmented reality as an orientation aid: a scavenger hunt prototype. In: 2015 International Conference on Intelligent environments (IE), pp 172–175

  191. Romero M, Usart M, Popescu M, Boyle E (2012) Interdisciplinary and international adaption and personalization of the metavals serious games. In: Serious Games Development and Applications, pp 59–73. Springer, Berlin

  192. Roto V (2006) Web browsing on mobile phones: characteristics of user experience. Helsinki University of Technology

  193. Rouanet P, Danieau F, Oudeyer PY (2011) A robotic game to evaluate interfaces used to show and teach visual objects to a robot in real world condition. In: Proceedings of the 6th international conference on Human-robot interaction, pp 313–320. ACM

  194. Rouse III R (2010) Game design: theory and practice. Jones & Bartlett Learning

  195. Rubio Ballester B, Bermúdez i Badia S, Verschure PF (2012) Including social interaction in stroke VR-based motor rehabilitation enhances performance: a pilot study. Presence Teleop Virt 21(4):490–501

    Article  Google Scholar 

  196. Ryan ML (Ed) (2004) Narrative across media: the languages of storytelling. U of Nebraska Press

  197. Salvador-Herranz G, Perez-Lopez D, Ortega M, Soto E, Alcaniz M, Contero M (2013) Manipulating virtual objects with your hands: a case study on applying desktop augmented reality at the primary school. In: 2013 46th Hawaii International Conference on System Sciences (HICSS), pp 31–39. IEEE

  198. Sánchez J (2003) End-user and facilitator questionnaire for Software Usability. Usability evaluation test. University of Chile, Santiago, Chile

    Google Scholar 

  199. Sánchez J (2012) Development of navigation skills through audio haptic videogaming in learners who are blind. Proc Comput Sci 14:102–110

    Article  Google Scholar 

  200. Sánchez J, Espinoza M (2011) Audio haptic videogaming for navigation skills in learners who are blind. In: The proceedings of the 13th international ACM SIGACCESS conference on Computers and accessibility, pp 227–228. ACM

  201. Sánchez J, Flores H, Sáenz M (2008) Mobile science learning for the blind. In: CHI‘08 extended abstracts on Human factors in computing systems, pp 3201–3206. ACM

  202. Sánchez J, Hassler T (2007) AudioMUD: a multiuser virtual environment for blind people. IEEE Trans Neural Syst Rehabil Eng 15(1):16–22

    Article  Google Scholar 

  203. Sánchez J, Sáenz M (2006) 3D sound interactive environments for blind children problem solving skills. Behav Inform Technol 25(4):367–378

    Article  Google Scholar 

  204. Sánchez J, Sáenz M, Garrido JM (2010) Usability of a multimodal video game to improve navigation skills for blind children. ACM Trans Access Comput (TACCESS) 3(2):7

    Google Scholar 

  205. Sánchez J, Salinas A, Sáenz M (2007) Mobile game-based methodology for science learning. In: Human-computer interaction. HCI applications and services, pp 322–331. Springer, Berlin

  206. Santos B, Romão T, Dias AE, Centieiro P (2013) eVision: a mobile game to improve environmental awareness. In: Advances in computer entertainment, pp 380–391. Springer International Publishing

  207. Scardovelli TA, Frère AF (2015) The design and evaluation of a peripheral device for use with a computer game intended for children with motor disabilities. Comput Methods Prog Biomed 118(1):44–58

    Article  Google Scholar 

  208. Scheible J, Tuulos VH, Ojala T (2007) Story Mashup: design and evaluation of novel interactive storytelling game for mobile and web users. In: Proceedings of the 6th international conference on Mobile and ubiquitous multimedia, pp 139–148. ACM

  209. Schell J (2008) The art of game design. Morgan Kaufmann

  210. Schmitz B, Klemke R, Specht M, Hoffmann M, Klamma R (2012) Developing a mobile game environment to support disadvantaged learners. In: 2012 I.E. 12th International Conference on Advanced Learning Technologies (ICALT), pp 223–227. IEEE

  211. Schneiderman B, Plaisant C (2004) Designing the user interface: strategies for effective human-computer interaction. Addison Wesley, New York, p 652

    Google Scholar 

  212. Schwebel DC, McClure LA, Severson J (2014) Usability and feasibility of an internet-based virtual pedestrian environment to teach children to cross streets safely. Virtual Reality 18(1):5–11

    Article  Google Scholar 

  213. Sharma SV, Shegog R, Chow J, Finley C, Pomeroy M, Smith C, Hoelscher DM (2015) Effects of the quest to lava mountain computer game on dietary and physical activity behaviors of elementary school children: a pilot group-randomized controlled trial. J Acad Nutr Diet

  214. Shegog R, Markham C, Peskin M, Dancel M, Coton C, Tortolero S (2007) ‘It’s Your Game’: An innovative multimedia virtual world to prevent HIV/STI and pregnancy in middle school youth. In Medinfo 2007: Proceedings of the 12th World Congress on Health (Medical) Informatics; Building Sustainable Health Systems, IOS Press, p 983

  215. Shepherd A (1989) Analysis and training in information technology tasks. Task analysis for human-computer interaction, pp 15–55

  216. Shin JH, Ryu H, Jang SH (2014) A task-specific interactive game-based virtual reality rehabilitation system for patients with stroke: a usability test and two clinical experiments. J Neuroeng Rehabil 11:32

    Article  Google Scholar 

  217. Song CR (2006) Design and Usability of Web-Based Serious Game for Taiwanese EFL Undergraduate Students. In: International Conference on Networking, International Conference on Systems and International Conference on Mobile Communications and Learning Technologies. ICN/ICONS/MCL 2006, pp 220–220. IEEE

  218. Song D, Karimi A, Kim P (2011) Toward designing mobile games for visually challenged children. In: 2011 International Conference on e-Education, Entertainment and e-Management (ICEEE), pp 234–238. IEEE

  219. Squire K (2005) Changing the game: what happens when video games enter the classroom? Innov: J Online Educ 1(6):5

    Google Scholar 

  220. Suárez M (2011) SIRIUS: Sistema de Evaluación de la Usabilidad Web Orientado al Usuario y basado en la Determinación de Tareas Críticas. Universidad de Oviedo, Oviedo

    Google Scholar 

  221. Suárez CO, Marco J, Baldassarri S, Cerezo E (2011) Children with special needs: comparing tactile and tangible interaction. In: Human-Computer Interaction-INTERACT 2011, pp 495–498. Springer, Berlin

  222. Sunwoo J, Yuen W, Lutteroth C, Wünsche B (2010) Mobile games for elderly healthcare. In: Proceedings of the 11th International Conference of the NZ Chapter of the ACM Special Interest Group on Human-Computer Interaction, pp 73–76. ACM

  223. Susaeta H, Jimenez F, Nussbaum M, Gajardo I, Andreu JJ, Villalta M (2010) From MMORPG to a classroom multiplayer presential role playing game. J Educ Technol Soc 13(3):257–269

    Google Scholar 

  224. Susi T, Johannesson M, Backlund P (2007) Serious games: an overview. Technical Report HS- IKI -TR-07-001. School of Humanities and Informatics, University of Skövde, Sweden

    Google Scholar 

  225. Sutcliffe A, Alrayes A (2012) Investigating user experience in Second Life for collaborative learning. Int J Hum Comput Stud 70(7):508–525

    Article  Google Scholar 

  226. Tadayon R, Panchanathan S, McDaniel T, Fakhri B, Laff M (2014) A toolkit for motion authoring and motor skill learning in serious games. In: 2014 I.E. International Symposium on Haptic, Audio and Visual Environments and Games (HAVE), pp 59–64. IEEE

  227. Tan JL, Goh DHL, Ang RP, Huan VS (2013) Participatory evaluation of an educational game for social skills acquisition. Comput Educ 64:70–80

    Article  Google Scholar 

  228. Tang TY, Man CY, Hang CP, Cheuk LS, Kwong CW, Chi YC, … Kam S (2008) A study of interaction patterns and awareness design elements in a massively multiplayer online game. Int J Comput Games Technol 2008: 5

  229. Taylor GS, Barnett JS (2012) Evaluation of wearable simulation interface for military training. Hum Factors: J Hum Factors Ergon Soc, 0018720812466892

  230. Thiele S, Meyer L, Geiger C, Drochtert D, Woldecke B (2013) Virtual archery with tangible interaction. In: 2013 I.E. Symposium on 3D User Interfaces (3DUI), pp 67–70. IEEE

  231. Tolentino GP, Battaglini C, Pereira ACV, de Oliveria RJ, de Paula MGM (2011) Usability of Serious Games for Health. In: 2011 Third International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES), pp 172–175. IEEE

  232. Tong T, Chignell M (2013) Designing game-based cognitive assessments for elderly adults. In: Proceedings of the First International Conference on Gameful design, research, and applications, pp 127–130. ACM

  233. Tong T, Chignell M (2014) Developing a serious game for cognitive assessment: choosing settings and measuring performance. In: Proceedings of the Second International Symposium of Chinese CHI, pp 70–79. ACM

  234. Torrente J (2012) Reusable game interfaces for people with disabilities. In: Proceedings of the 14th international ACM SIGACCESS conference on Computers and accessibility, pp 301–302. ACM

  235. Trewin S, Laff M, Hanson V, Cavender A (2009) Exploring visual and motor accessibility in navigating a virtual world. ACM Trans Access Comput (TACCESS) 2(2):11

    Google Scholar 

  236. Tullis TS, Stetson JN (2004) A comparison of questionnaires for assessing website usability. In: Usability Professional Association Conference, pp 1–12

  237. Vajda P, Ivanov I, Lee JS, Ebrahimi T (2011) Epitomize your photos. Int J Comput Games Technol 2011:9

    Article  Google Scholar 

  238. Valdes B, Hilderman CG, Hung CT, Shirzad N, Van der Loos HF (2014) Usability testing of gaming and social media applications for stroke and cerebral palsy upper limb rehabilitation. In: 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp 3602–3605. IEEE

  239. Villalta M, Gajardo I, Nussbaum M, Andreu JJ, Echeverría A, Plass JL (2011) Design guidelines for classroom multiplayer presential games (CMPG). Comput Educ 57(3):2039–2053

    Article  Google Scholar 

  240. Virvou M, Katsionis G (2008) On the usability and likeability of virtual reality games for education: the case of VR-ENGAGE. Comput Educ 50(1):154–178

    Article  Google Scholar 

  241. Virzi RA (1992) Refining the test phase of usability evaluation: how many subjects is enough? Hum Factors: J Hum Factors Ergon Soc 34(4):457–468

    Google Scholar 

  242. Wallner G, Kriglstein S (2011) Design and evaluation of the educational game DOG eometry: a case study. In: Proceedings of the 8th International Conference on Advances in Computer Entertainment Technology, p 14. ACM

  243. Wang AI, Morch-Storstein OK (2008) An evaluation of a mobile game concept for lectures. In: IEEE 21st Conference on Software Engineering Education and Training. CSEET‘08, pp 197–204. IEEE

  244. Wang AI, Wu B, Bakken SK (2010) Experiences from implementing a face-to-face educational game for iPhone/iPod Touch. InGames Innovations Conference (ICE-GIC), 2010 International IEEE Consumer Electronics Society’s, pp 1–8. IEEE

  245. Weightman APH, Preston N, Holt R, Allsop M, Levesley M, Bhakta B (2010) Engaging children in healthcare technology design: developing rehabilitation technology for children with cerebral palsy. J Eng Des 21(5):579–600

    Article  Google Scholar 

  246. Wharton C, Rieman J, Lewis C, Polson P (1994) The cognitive walkthrough method: a practitioner’s guide. In: Usability inspection methods, pp 105–140. John Wiley & Sons, Inc.

  247. Whitlock LA, McLaughlin AC, Leidheiser W, Gandy M, Allaire JC (2014) Know before you go: feelings of flow for older players depends on game and player characteristics. In: Proceedings of the first ACM SIGCHI annual symposium on Computer-human interaction in play, pp 277–286. ACM

  248. Wise K, Kobayashi K, Searchfield GD (2015) Feasibility study of a game integrating assessment and therapy of tinnitus. J Neurosci Methods 249:1–7

    Article  Google Scholar 

  249. Wrońska N, Garcia-Zapirain B, Mendez-Zorrilla A (2015) An iPad-Based tool for improving the skills of children with attention deficit disorder. Int J Environ Res Public Health 12(6):6261–6280

    Article  Google Scholar 

  250. Wu W, Rank S (2015) Responsive environmental Diegetic audio feedback for hand gestures in audio-only games. In: Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play, pp 739–744. ACM

  251. Yamabe T, Asuma H, Kiyono S, Nakajima T (2011) Feedback design in augmented musical instruments: A case study with an ar drum kit. In: 2011 I.E. 17th International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA) (Vol. 2), pp 126–129. IEEE

  252. Yáñez Gómez R, Cascado Caballero D, Sevillano JL (2014) Heuristic evaluation on mobile interfaces: a new checklist. Sci World J 2014

  253. Yang HC, Zapata-Rivera D (2010) Interlanguage pragmatics with a pedagogical agent: the request game. Comput Assist Lang Learn 23(5):395–412

    Article  Google Scholar 

  254. Yoon JW, Yang SI, Cho SB (2012) Adaptive mixture-of-experts models for data glove interface with multiple users. Expert Syst Appl 39(5):4898–4907

    Article  Google Scholar 

  255. Youm S (2015) The development and user satisfaction evaluation of Internet-Based N-screen healthcare walking content to increase continuous usage motivation. Telemedicine and e-Health

  256. Yun C, Shastri D, Pavlidis I, Deng Z (2009) O’game, can you feel my frustration?: improving user's gaming experience via stresscam. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp 2195–2204. ACM

  257. Zhang D. Overview of usability evaluation methods. http://www.usabilityhome.com. Retrieved in March 2014

  258. Zhang M, Xu M, Han L, Liu Y, Lv P, He G (2012) Virtual Network Marathon with immersion, scientificalness, competitiveness, adaptability and learning. Comput Graph 36(3):185–192

    Article  Google Scholar 

  259. Zin NAM, Nasir NYM (2008) Evaluation of an edutainment animated folktales software to motivate socio-cultural awareness among children. In: Third International Conference on Convergence and Hybrid Information Technology, 2008. ICCIT‘08. (Vol. 1), pp 315–319. IEEE

Download references

Acknowledgments

This work has been supported by project PROCUR@-IPT-2011-1038-900000, funded by the program INNPACTO of MINECO and FEDER funds; and by the Telefonica Chair “Intelligence in Networks” of the Universidad de Sevilla, Spain.

Author information

Authors and Affiliations

  1. Department of Computer Technology and Architecture, ETS Ingenieria Informatica, Universidad de Sevilla, Avda. Reina Mercedes s/n., 41012, Seville, Spain

    Rosa Yáñez-Gómez, Daniel Cascado-Caballero & José-Luis Sevillano

Authors
  1. Rosa Yáñez-Gómez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel Cascado-Caballero
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. José-Luis Sevillano
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rosa Yáñez-Gómez.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yáñez-Gómez, R., Cascado-Caballero, D. & Sevillano, JL. Academic methods for usability evaluation of serious games: a systematic review. Multimed Tools Appl 76, 5755–5784 (2017). https://doi.org/10.1007/s11042-016-3845-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-016-3845-9

Keywords

Search

Navigation