Development of an Audio-Haptic Virtual Interface for Navigation of Large-Scale Environments for People Who Are Blind

  • Lotfi B. MerabetEmail author
  • Jaime Sánchez
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9739)


We are investigating cognitive spatial mapping skills in people who are blind through the use of virtual navigation and assessing the transference of acquired spatial knowledge in large-scale, real-world navigation tasks. Training is carried out with a user-centered, computer-based, navigation software platform called Haptic Audio Game Application (HAGA). This software was developed to assist in orientation and mobility (O&M) training by introducing blind users to a spatial layout of a large-scale environment through immersive and simulation-based virtual navigation. As part of a self-directed, free exploration strategy, users interact with HAGA in order to navigate through a simulated indoor and outdoor virtual environment that represents an actual physical space. Navigation is based on the use of iconic and spatialized auditory cues and vibro-tactile feedback so as to build a cognitive spatial map of the surrounding environment. The ability to transfer acquired spatial information is then assessed in a series of physical navigation tasks carried out in the actual target environment explored virtually.


Multimodal interfaces Blind Spatial cognition Navigation 



This work was supported by an NIH/NEI RO1 GRANT EY019924 (Lotfi B. Merabet) and also funded by the Chilean National Fund for Scientific and Technological Development, Fondecyt HYPERLINK “tel:1150898” 1150898; and the Basal Funds for Centers of Excellence, FB0003 project, from the Associative Research Program of CONICYT, Chile. (Jaime Sánchez). The authors would like to thank the research participants, as well as Rabih Dow, Padma Rajagopal and the staff of the Carroll Center for the Blind (Newton MA, USA) for their support in carrying out this research.


  1. 1.
    Blasch, B.B., Wiener, W.R., Welsh, R.L.: Foundations of Orientation and Mobility, 2nd ed., xx, 775 p. AFB Press, New York (1997)Google Scholar
  2. 2.
    Landau, B., Gleitman, H., Spelke, E.: Spatial knowledge and geometric representation in a child blind from birth. Science 213(4513), 1275–1278 (1981)CrossRefGoogle Scholar
  3. 3.
    Strelow, E.R.: What is needed for a theory of mobility: direct perception and cognitive maps–lessons from the blind. Psychol. Rev. 92(2), 226–248 (1985)CrossRefGoogle Scholar
  4. 4.
    Tolman, E.C.: Cognitive maps in rats and men. Psychol. Rev. 55(4), 189–208 (1948)CrossRefGoogle Scholar
  5. 5.
    Ashmead, D.H., Hill, E.W., Talor, C.R.: Obstacle perception by congenitally blind children. Percept. Psychophys. 46(5), 425–433 (1989)CrossRefGoogle Scholar
  6. 6.
    Ashmead, D.H., et al.: Spatial hearing in children with visual disabilities. Perception 27(1), 105–122 (1998)CrossRefGoogle Scholar
  7. 7.
    Thinus-Blanc, C., Gaunet, F.: Representation of space in blind persons: vision as a spatial sense? Psychol. Bull. 121(1), 20–42 (1997)CrossRefGoogle Scholar
  8. 8.
    Pasqualotto, A., Proulx, M.J.: The role of visual experience for the neural basis of spatial cognition. Neurosci. Biobehav. Rev. 36(4), 1179–1187 (2012)CrossRefGoogle Scholar
  9. 9.
    Axelrod, S.: Effects of early blindness; performance of blind and sighted children on tactile and auditory tasks, ix, 83 p. American Foundation for the Blind, New York (1959)Google Scholar
  10. 10.
    Loomis, J.M., Klatzky, R.L., Golledge, R.G.: Navigating without vision: basic and applied research. Optom. Vis. Sci. 78(5), 282–289 (2001)CrossRefGoogle Scholar
  11. 11.
    Fortin, M., et al.: Wayfinding in the blind: larger hippocampal volume and supranormal spatial navigation. Brain 131(Pt 11), 2995–3005 (2008)CrossRefGoogle Scholar
  12. 12.
    Bavelier, D., Green, C.S., Dye, M.W.: Children, wired: for better and for worse. Neuron 67(5), 692–701 (2010)CrossRefGoogle Scholar
  13. 13.
    Bavelier, D., et al.: Brains on video games. Nat. Rev. Neurosci. 12(12), 763–768 (2011)CrossRefGoogle Scholar
  14. 14.
    Dede, C.: Immersive interfaces for engagement and learning. Science 323(5910), 66–69 (2009)CrossRefGoogle Scholar
  15. 15.
    Shaffer, D.W., et al.: Video games and the future of learning. Phi Delta Kappan 87, 104–111 (2005)CrossRefGoogle Scholar
  16. 16.
    Lange, B., et al.: Designing informed game-based rehabilitation tasks leveraging advances in virtual reality. Disabil. Rehabil. 34, 1863–1870 (2012)CrossRefGoogle Scholar
  17. 17.
    Mayo, M.J.: Video games: a route to large-scale STEM education? Science 323(5910), 79–82 (2009)CrossRefGoogle Scholar
  18. 18.
    Giudice, N.A., Bakdash, J.Z., Legge, G.E.: Wayfinding with words: spatial learning and navigation using dynamically updated verbal descriptions. Psychol. Res. 71(3), 347–358 (2007)CrossRefGoogle Scholar
  19. 19.
    Ohuchi, M., et al.: Cognitive-map formation of blind persons in a virtual sound environment. In: Proceedings of the 12th International Conference on Auditory Display, London, UK (2006)Google Scholar
  20. 20.
    Riehle, T.H., Lichter, P., Giudice, N.A.: An indoor navigation system to support the visually impaired. Conf. Proc. IEEE Eng. Med. Biol. Soc. 1, 4435–4438 (2008)Google Scholar
  21. 21.
    Legge, G.E., et al.: Indoor navigation by people with visual impairment using a digital sign system. PLoS ONE 8(10), e76783 (2013)CrossRefGoogle Scholar
  22. 22.
    Johnson, L.A., Higgins, C.M.: A navigation aid for the blind using tactile-visual sensory substitution. Conf. Proc. IEEE Eng. Med. Biol. Soc. 1, 6289–6292 (2006)Google Scholar
  23. 23.
    Lahav, O.: Using virtual environment to improve spatial perception by people who are blind. Cyberpsychol. Behav. 9(2), 174–177 (2006)CrossRefGoogle Scholar
  24. 24.
    Pissaloux, E., et al.: Space cognitive map as a tool for navigation for visually impaired. Conf. Proc. IEEE Eng. Med. Biol. Soc. 1, 4913–4916 (2006)Google Scholar
  25. 25.
    Driver, J., Noesselt, T.: Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments. Neuron 57(1), 11–23 (2008)CrossRefGoogle Scholar
  26. 26.
    Sanchez, J., Lumbreras, M.: 3D aural interactive hyperstories for blind children. Int. J. Virtual Reality 4(1), 20–28 (1998)Google Scholar
  27. 27.
    Connors, E.C., et al.: Virtual environments for the transfer of navigation skills in the blind: a comparison of directed instruction vs. video game based learning approaches. Front. Hum. Neurosci. 8, 223 (2014)Google Scholar
  28. 28.
    Connors, E.C., et al.: Action video game play and transfer of navigation and spatial cognition skills in adolescents who are blind. Front. Hum. Neurosci. 8, 133 (2014)Google Scholar
  29. 29.
    Connors, E.C., et al.: Development of an audio-based virtual gaming environment to assist with navigation skills in the blind. J. Visualized Exp. JoVE 73 (2013)Google Scholar
  30. 30.
    Merabet, L.B., et al.: Teaching the blind to find their way by playing video games. PLoS ONE 7(9), e44958 (2012)CrossRefGoogle Scholar
  31. 31.
    Halko, M.A., et al.: Real world navigation independence in the early blind correlates with differential brain activity associated with virtual navigation. Hum. Brain Mapp. 35(6), 2768–2778 (2014)CrossRefGoogle Scholar
  32. 32.
    Shneiderman, B., Plaisant, C., Jacobs, S.: Designing the User Interface: Strategies for Effective Human-Computer Interaction, 5th edn. Addison Wesley, Boston (2009)Google Scholar
  33. 33.
    Merabet, L.B., Pascual-Leone, A.: Neural reorganization following sensory loss: the opportunity of change. Nat. Rev. Neurosci. 11(1), 44–52 (2010)CrossRefGoogle Scholar
  34. 34.
    Beggs, W.D.: Psychological correlates of walking speed in the visually impaired. Ergonomics 34(1), 91–102 (1991)CrossRefGoogle Scholar
  35. 35.
    Soong, G.P., Lovie-Kitchin, J.E., Brown, B.: Preferred walking speed for assessment of mobility performance: sighted guide versus non-sighted guide techniques. Clin. Exp. Optom. 83(5), 279–282 (2000)CrossRefGoogle Scholar
  36. 36.
    Hartong, D.T., et al.: Improved mobility and independence of night-blind people using night-vision goggles. Invest. Ophthalmol. Vis. Sci. 45(6), 1725–1731 (2004)CrossRefGoogle Scholar
  37. 37.
    Kalia, A.A., et al.: Assessment of indoor route-finding technology for people with visual impairment. J. Vis. Impairment Blindness 104(3), 135–147 (2010)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.Laboratory for Visual Neuroplasticity, Massachusetts Eye and Ear InfirmaryHarvard Medical SchoolBostonUSA
  2. 2.Department of Computer ScienceUniversity of ChileSantiagoChile

Personalised recommendations