Multimodal Videogames for the Cognition of People Who Are Blind: Trends and Issues

  • Jaime Sánchez
  • Ticianne DarinEmail author
  • Rossana Andrade
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9177)


Multimodal serious games are attractive tools for achieving this goal and helping people with visual disabilities to perceive and to interpret the surrounding world. However, it is fundamental to ensure that the games can stimulate cognitive development. The purpose of this study was to investigate the role of multimodal components in the development and evaluation of games and virtual environments targeting the enhancement of cognitive skills in people who are blind. We analyze the state-of-the-art concerning approaches and technologies currently in use for the development of mental maps, cognitive spatial structures, and navigation skills in learners who are blind by using multimodal videogames. Besides, we identify the current approaches used for designing and evaluating multimodal games in this context. In this paper, we discuss the results on these and related topics and draw from them some trends and issues.


Accessible games Multimodal interfaces Cognition Blind people 



This report was funded by the Program STIC-AmSud-CAPES/CONYCIT/MAEE, project KIGB-Knowing and Interacting while Gaming for the Blind, 2014.


  1. 1.
    Kolb, B., Whishaw, I.: Neuropsicología Humana 5a edición. In: Editorial Médica Panamericana 2006, p. 555 (2006)Google Scholar
  2. 2.
    Pressl, B., Wieser, M.: A computer-based navigation system tailored to the needs of blind people. In: Miesenberger, K., Klaus, J., Zagler, W.L., Karshmer, A.I. (eds.) ICCHP 2006. LNCS, vol. 4061, pp. 1280–1286. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  3. 3.
    Kulyukin, V., Gharpure, C., Nicholson, J., Pavithran, S.: FID in robot-assisted indoor navigation for the visually impaired. In: IEEE/RSJ Intelligent Robots and Systems (IROS 2004) Conference, September - October 2004, Sendai Kyodo Printing: Sendai, Japan, pp. 1979–1984 (2004)Google Scholar
  4. 4.
    Sánchez, J., Elias, M.: Guidelines for designing mobility and orientation software for blind children. In: Baranauskas, C., Abascal, J., Barbosa, S.D.J. (eds.) INTERACT 2007. LNCS, vol. 4662, pp. 375–388. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  5. 5.
    Loomis, J.M., Klatzky, R.L., Golledge, R.G.: Navigating without vision: Basic and applied research. Optom. Vis. Sci. 78(5), 282–289 (2001)CrossRefGoogle Scholar
  6. 6.
    Lahav, O., Mioduser, D.: Haptic-feedback support for cognitive mapping of unknown spaces by people who are blind. Int. J. Hum.-Comput. Stud. 66, 23–35 (2008)CrossRefGoogle Scholar
  7. 7.
    Sánchez, J., Zúñiga, M.: Evaluating the Interaction of Blind Learners with Audio-Based Virtual Environments. Cybersychol. Behav. 9(6), 717 (2006)Google Scholar
  8. 8.
    Lahav, O., Schloerb, D.W., Kumar, S., Srinivasan, M.A.: BlindAid: A learning environment for enabling people who are blind to explore and navigate through unknown real spaces. In: Proceedings Virtual Rehabilitation 2008 Conference, Vancouver, Canada, pp. 193–197 (2008)Google Scholar
  9. 9.
    Sánchez, J., Tadres, A.: Audio and haptic based virtual environments for orientation and mobility in people who are blind. In: Proceedings of the 12th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2010), pp. 237–238. ACM, New York (2010)Google Scholar
  10. 10.
    Mayo, M.: Games for science and engineering education. Commun. ACM 30(35), 30–35 (2007)CrossRefGoogle Scholar
  11. 11.
    Yuan, B.: Towards generalized accessibility of video games for the visually impaired. Ph.D. Dissertation. Univ. of Nevada, Reno, Reno, NV, USA. Advisor (S Frederick C. Harris and Eelke Folmer. AAI3355610) (2009)Google Scholar
  12. 12.
    Yuan, B., Folmer, E.: Blind hero: enabling guitar hero for the visually impaired. In: Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility, pp 69–176 (2008)Google Scholar
  13. 13.
    Westin, T.: Game accessibility case study: terraformers – a real-time 3D graphic game. In: Proceedings of the 5th International Conference on Disability, Virtual Reality and Associated Technologies, ICDVRAT 2004, Oxford, UK, pp. 95–100 (2004)Google Scholar
  14. 14.
    Petersen, K., Feldt, R., Mujtaba, S., Mattsson, M.: Systematic mapping studies in software engineering. In: EASE 2008, Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering, University of Bari, Italy (2008)Google Scholar
  15. 15.
    Kitchenham, B., Charters, S.: Guidelines for performing systematic literature reviews in software engineering. Technical report EBSE 2007–001, Keele University and Durham University Joint Report (2007)Google Scholar
  16. 16.
    Fabbri, S., Hernandes, E., Di Thommazo, A., Belgamo, A., Zamboni, A., Silva, C.: Managing literature reviews information through visualization. In 14th International Conference on Enterprise Information Systems, ICEIS, Wroclaw, Poland, June 2012. Scitepress, Lisbon (2012)Google Scholar
  17. 17.
    Lewis-Beck, M.S., Bryman, A., Liao, T.F.: The SAGE Encyclopedia of Social Science Research Methods (2004)Google Scholar
  18. 18.
    Connors, E.C., Chrastil, E.R., Sánchez, J., Merabet, L.B.: Action video game play and transfer of navigation and spatial cognition skills in adolescents who are blind. Frontiers in Human Neuroscience (FNHUM) 8, article 133, Frontiers (2014)Google Scholar
  19. 19.
    Sánchez, J., Saenz, M., Garrido, J.M.: Usability of a multimodal video game to improve navigation skills for blind children. In: ACM Transactions on Accessible Computing, Proceedings of the 11th International ACM SIGACCESS Conference on Computers End Accessibility, pp. 35–42 (2010)Google Scholar
  20. 20.
    Dulyan, A., Edmonds, E.: AUXie: initial evaluation of a blind-accessible virtual museum tour. In: Proceedings of the 22nd Australasian Computer-Human Interaction Conference, Brisbane, Australia, pp. 272–275 (2010)Google Scholar
  21. 21.
    Espinoza, M., Sánchez, J., Campos, M.B.: Videogaming interaction for mental model construction in learners who are blind. In: Proceedings of 8th International Conference, UAHCI 2014, Held as Part of HCI International, Crete, Greece, pp. 525–536 (2014)Google Scholar
  22. 22.
    Guerrero, J., Lincon, J.: AINIDIU, CANDI, HELPMI: ICTs of a personal experience. In: 2012 Workshop on Engineering Applications (WEA), Bogotá, Colombia, pp. 1–7 (2012)Google Scholar
  23. 23.
    Lahav, O, Schloerb, D.W., Srinivasan, M.A.: Virtual environment system in support of a traditional orientation and mobility rehabilitation program for people who are blind. In: Presence, vol. 22, pp. 235–254. MIT Press (2013)Google Scholar
  24. 24.
    Lahav, O., Mioduser, D.: Construction of cognitive maps of unknown spaces using a multysensory virtual environment for people who are blind. Comput. Hum. Behav. 24(3), 1139–1155 (2008)CrossRefGoogle Scholar
  25. 25.
    Lumbreras, M., Sánchez, J.: Interactive 3D sound hyperstories for blind children. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Pittsburgh, USA, pp. 318–325, May 1999Google Scholar
  26. 26.
    Mccrindle, R.J., Symons, D.: Audio space invaders. In: Proceedings of the Third International Conference on Disability, Virtual Reality and Associated Technologies, pp. 59–65 (2000)Google Scholar
  27. 27.
    Sánchez, J.: A model to design interactive learning environments for children with visual disabilities. In: Education and Information Technologies, vol. 12(3), pp. 149–163. Kluwer Academic Publishers-Plenum Publishers (2007)Google Scholar
  28. 28.
    Sánchez, J.: Development of navigation skills through audio haptic videogaming in learners who are blind. In: Proceedings of the Software Development for Enhancing Accessibility and Fighting Info-Exclusion (DSAI), July 2012. Douro, Portugal, pp. 102–110 (2012)Google Scholar
  29. 29.
    Sánchez, J., Aguayo, F.: AudioGene: mobile learning genetics through audio by blind learners. In: Kendall, M., Samways, B. (eds.) Learning to Live in the Knowledge Society. IFIP, vol. 281, pp. 79–86. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  30. 30.
    Sánchez, J., Campos, M.B., Espinoza, M., Merabet, L.B.: Audio haptic videogaming for developing wayfinding skills in learners who are blind. In: Proceedings of the ACM International Conference on Intelligent User Interfaces (ACM IUI), February 2014, Haifa, Israel, pp. 199–208 (2014)Google Scholar
  31. 31.
    Sánchez, J., Espinoza, M.: Designing serious videogames through concept maps. In: Kurosu, M. (ed.) HCII/HCI 2013, Part II. LNCS, vol. 8005, pp. 299–308. Springer, Heidelberg (2013)Google Scholar
  32. 32.
    Sánchez, J., Espinoza, M., Campos, M.B., Leite, L.L.: Modeling videogames for mental mapping in people who are blind. In: Proceedings of 8th International Conference, UAHCI 2014, Held as Part of HCI International 2014, Crete, Greece, pp. 605–616 (2014)Google Scholar
  33. 33.
    Sánchez, J., Espinoza, M., Campos, M.B., Merabet, L.B.: Enhancing orientation and mobility skills in learners who are blind through videogaming. In: Proceedings ACM Creativity and Cognition (ACM C&C), June 2013, Sydney, Australia, pp. 353–356 (2013)Google Scholar
  34. 34.
    Sánchez, J., Guerrero, L., Sáenz, M., Flores, H.: A model to develop videogames for orientation and mobility. In: Miesenberger, K., Klaus, J., Zagler, W., Karshmer, A. (eds.) ICCHP 2010, Part II. LNCS, vol. 6180, pp. 296–303. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  35. 35.
    Sánchez, J., Mascaró, J.: Audiopolis, navigation through a virtual city using audio and haptic interfaces for people who are blind. In: Stephanidis, C. (ed.) Universal Access in HCI, Part II, HCII 2011. LNCS, vol. 6766, pp. 362–371. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  36. 36.
    Sánchez, J., Flores, H.: Virtual mobile science learning for blind people. Cyberpsychology & Behavior: the Impact of the Internet, Multimedia and Virtual Reality on Behavior and Society, vol. 11(3) (2008)Google Scholar
  37. 37.
    Sánchez, J., Sáenz, M.: 3D sound interactive environments for blind children problem solving skills. In: Behaviour & IT. From the 7th International ACM SIGACCESS Conference on Computers and Accessibility, Baltimore, USA, vol. 25(4), pp. 367–378 (2006)Google Scholar
  38. 38.
    Sánchez, J., Sáenz, M.: Video gaming for blind learners school integration in science classes. In: Gross, T., Gulliksen, J., Kotzé, P., Oestreicher, L., Palanque, P., Prates, R.O., Winckler, M. (eds.) INTERACT 2009. LNCS, vol. 5726, pp. 36–49. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  39. 39.
    Sánchez, J., Sáenz, M.: Metro navigation for the blind. Computers and Education (CAE) 55(3), 970–981 (2010) (Elsevier Science, Amsterdam, The Netherlands)Google Scholar
  40. 40.
    Sánchez, J., Tadres, A., Pascual-Leone, A., Merabet, L.: Blind children navigation through gaming and associated brain plasticity. In: Virtual Rehabilitation International Conference 2009, June 29–July 2, 2004, Haifa, Israel, pp. 29–36 (2009)Google Scholar
  41. 41.
    Sánchez, J., Sáenz, M., Pascual-Leone, A., Merabet, L.B. Enhancing navigation skills through audio gaming. In: Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI), April 2010, Atlanta, GA, USA, pp. 3991–3996 (2010)Google Scholar
  42. 42.
    Sánchez, J., Sáenz, M.: Three-dimensional virtual environments for blind children. CyberPsychol. Behav. 9(2), 200 (2006)CrossRefGoogle Scholar
  43. 43.
    Torrente, J., Blanco, Á., Serrano-Laguna, Á., Vallejo-Pinto, J.Á., Moreno-Ger, P., Fernández-Manjón, B.: Towards a low cost adaptation of educational games for people with disabilities. Comput. Sci. Inf. Syst. 11(1), 369–391 (2014)CrossRefGoogle Scholar
  44. 44.
    Torrente, J., Blanco, A., Moreno-Ger, P., Martinez-Ortiz, I., Fernandez-Manjon, B.: Implementing accessibility in educational videogames with e-adventure. In: Proceedings of the 1st ACM International Workshop on Multimedia Technologies for Distance Learning, Beijing, China, pp. 57–66 (2009)Google Scholar
  45. 45.
    Trewin, S., Hanson, V.L., Laff, M.R., Cavender, A.: PowerUp: an accessible virtual world. In: Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility, Halifax, Canada, pp. 177–184 (2008)Google Scholar
  46. 46.
    Sanchez, J.: Software Usability for Blind Children Questionnaire (SUBC). Usability evaluation test, University of Chile (2003)Google Scholar
  47. 47.
    Sánchez, J.: End-user and facilitator questionnaire for Software Usability. Usability evaluation test. University of Chile (2003)Google Scholar
  48. 48.
    Alba, J.: Introducción a las Interfaces Multimodales. Revista Bit, No.157, 2006, pp. 56–58. Disponible en (2006). (Last Accessed 13 November 2014)

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Jaime Sánchez
    • 1
  • Ticianne Darin
    • 2
    Email author
  • Rossana Andrade
    • 3
  1. 1.Department of Computer ScienceUniversity of ChileSantiagoChile
  2. 2.Virtual University InstituteFederal University of CearáFortalezaBrazil
  3. 3.Department of Computer ScienceFederal University of CearáFortalezaBrazil

Personalised recommendations