Experimental Brain Research

, Volume 233, Issue 3, pp 851–860 | Cite as

Using an auditory sensory substitution device to augment vision: evidence from eye movements

  • Thomas D. Wright
  • Aaron Margolis
  • Jamie Ward
Research Article


Sensory substitution devices convert information normally associated with one sense into another sense (e.g. converting vision into sound). This is often done to compensate for an impaired sense. The present research uses a multimodal approach in which both natural vision and sound-from-vision (‘soundscapes’) are simultaneously presented. Although there is a systematic correspondence between what is seen and what is heard, we introduce a local discrepancy between the signals (the presence of a target object that is heard but not seen) that the participant is required to locate. In addition to behavioural responses, the participants’ gaze is monitored with eye-tracking. Although the target object is only presented in the auditory channel, behavioural performance is enhanced when visual information relating to the non-target background is presented. In this instance, vision may be used to generate predictions about the soundscape that enhances the ability to detect the hidden auditory object. The eye-tracking data reveal that participants look for longer in the quadrant containing the auditory target even when they subsequently judge it to be located elsewhere. As such, eye movements generated by soundscapes reveal the knowledge of the target location that does not necessarily correspond to the actual judgment made. The results provide a proof of principle that multimodal sensory substitution may be of benefit to visually impaired people with some residual vision and, in normally sighted participants, for guiding search within complex scenes.


Sensory substitution Blindness Multisensory Auditory Eye-tracking Spatial 


  1. Arnott SR, Alain C (2011) The auditory dorsal pathway: orienting vision. Neurosci Biobehav Rev 35:2162–2173. doi: 10.1016/j.neubiorev.2011.04.005 PubMedCrossRefGoogle Scholar
  2. Brown DJ, Macpherson T, Ward J (2011) Seeing with sound? Exploring different characteristics of a visual-to-auditory sensory substitution device. Perception 40:1120–1135PubMedCrossRefGoogle Scholar
  3. Capelle C, Trullemans C, Arno P, Veraart C (1998) A real-time experimental prototype for enhancement of vision rehabilitation using auditory substitution. IEEE Trans Biomed Eng 45:1279–1293. doi: 10.1109/10.720206 PubMedCrossRefGoogle Scholar
  4. Colonius H, Arndt P (2001) A two-stage model for visual–auditory interaction in saccadic latencies. Percept Psychophys 63:126–147. doi: 10.3758/BF03200508 PubMedCrossRefGoogle Scholar
  5. Corneil BD, Wanrooij MV, Munoz DP, Opstal AJV (2002) Auditory–visual interactions subserving goal-directed saccades in a complex scene. J Neurophysiol 88:438–454PubMedGoogle Scholar
  6. Deneve S, Pouget A (2004) Bayesian multisensory integration and cross-modal spatial links. J Physiol. doi: 10.1016/j.jphysparis.2004.03.011 Google Scholar
  7. Frassinetti F, Pavani F, Ladavas E (2002) Acoustical vision of neglected stimuli: interaction among spatially converging audiovisual inputs in neglect patients. J Cogn Neurosci 14:62–69PubMedCrossRefGoogle Scholar
  8. Grothe B, Pecka M, McAlpine D (2010) Mechanisms of sound localization in mammals. Physiol Rev 90:983–1012. doi: 10.1152/physrev.00026.2009 PubMedCrossRefGoogle Scholar
  9. Haigh A, Brown DJ, Meijer P, Proulx MJ (2013) How well do you see what you hear? The acuity of visual-to-auditory sensory substitution. Front Psychol. doi: 10.3389/fpsyg.2013.00330 PubMedCentralPubMedGoogle Scholar
  10. Hamilton-Fletcher G, Ward J (2013) Representing colour through hearing and touch in sensory substitution devices. Multisens Res 26:503–532. doi: 10.1163/22134808-00002434 PubMedCrossRefGoogle Scholar
  11. Hancock PA, Mercado JE, Merlo J, Van Erp JBF (2013) Improving target detection in visual search through the augmenting multi-sensory cues. Ergonomics 56:729–738. doi: 10.1080/00140139.2013.771219 PubMedCrossRefGoogle Scholar
  12. Kastner S, De Weerd P, Pinsk MA, Elizondo MI, Desimone R, Ungerleider LG (2001) Modulation of sensory suppression: implications for receptive field sizes in the human visual cortex. J Neurophysiol 86:1398–1411PubMedGoogle Scholar
  13. Kim J-K, Zatorre RJ (2008) Generalized learning of visual-to-auditory substitution in sighted individuals. Brain Res 242:263–275CrossRefGoogle Scholar
  14. Klapetek A, Ngo MK, Spence C (2012) Does crossmodal correspondence modulate the facilitatory effect of auditory cues on visual search? Atten Percept Psychophys 74:1154–1167. doi: 10.3758/s13414-012-0317-9 PubMedCrossRefGoogle Scholar
  15. Levy-Tzedek S, Hanassy S, Abboud S, Maidenbaum S, Amedi A (2012a) Fast, accurate reaching movements with a visual-to-auditory sensory substitution device. Restor Neurol Neurosci 30:313–323. doi: 10.3233/rnn-2012-110219 PubMedGoogle Scholar
  16. Levy-Tzedek S, Novick I, Arbel R, Abboud S, Maidenbaum S, Vaadia E, Amedi A (2012b) Cross-sensory transfer of sensory-motor information: visuomotor learning affects performance on an audiomotor task, using sensory-substitution. Sci Rep. doi: 10.1038/srep00949 PubMedCentralPubMedGoogle Scholar
  17. Mannan SK, Pambakian ALM, Kennard C (2010) Compensatory strategies following visual search training in patients with homonymous hemianopia: an eye movement study. J Neurol 257:1812–1821PubMedCentralPubMedCrossRefGoogle Scholar
  18. Matusz PJ, Eimer M (2013) Top-down control of audiovisual search by bimodal search templates. Psychophysiology 50:996–1009. doi: 10.1111/psyp.12086 Google Scholar
  19. Meijer PBL (1992) An experimental system for auditory image representations. IEEE Trans Biomed Eng 39:112–121PubMedCrossRefGoogle Scholar
  20. Meredith MA, Stein BE (1986) Visual, auditory and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol 56:640–662PubMedGoogle Scholar
  21. Orchard-Mills E, Van der Burg E, Alais D (2013) Amplitude-modulated auditory stimuli influence selection of visual spatial frequencies. J Vis. doi: 10.1167/13.3.6 PubMedGoogle Scholar
  22. Pavani F, Ladavas E, Driver J (2003) Auditory and multisensory aspects of visuospatial neglect. Trends Cogn Sci 7:407–414PubMedCrossRefGoogle Scholar
  23. Poirier C, De Volder AG, Tranduy D, Scheiber C (2007) Pattern recognition using a device substituting audition for vision in blindfolded sighted subjects. Neuropsychologia 45:1108–1121PubMedCrossRefGoogle Scholar
  24. Proulx MJ, Ptito M, Amedi A (2014) Multisensory integration, sensory substitution and visual rehabilitation. Neurosci Biobehav Rev 41:1–2. doi: 10.1016/j.neubiorev.2014.03.004 PubMedCrossRefGoogle Scholar
  25. Renier L, Collignon O, Poirier C et al (2005) Cross-modal activation of visual cortex during depth perception using auditory substitution of vision. Neuroimage 26:573–580. doi: 10.1016/j.neuroimage.2005.01.047 PubMedCrossRefGoogle Scholar
  26. Spence C (2011) Crossmodal correspondences: a tutorial review. Atten Percept Psychophys 73:971–995. doi: 10.3758/s13414-010-0073-7 PubMedCrossRefGoogle Scholar
  27. Van der Burg E, Olivers CNL, Bronkhorst AW, Theeuwes J (2008) Pip and pop: nonspatial auditory signals improve spatial visual search. J Exp Psychol Hum Percep Perform 34:1053–1065. doi: 10.1037/0096-1523.34.5.1053 CrossRefGoogle Scholar
  28. Van der Burg E, Olivers CNL, Bronkhorst AW, Theeuwes J (2009) Poke and pop: tactile–visual synchrony increases visual saliency. Neurosci Lett 450:60–64. doi: 10.1016/j.neulet.2008.11.002 PubMedCrossRefGoogle Scholar
  29. Ward J, Meijer P (2010) Visual experiences in the blind induced by an auditory sensory substitution device. Conscious Cogn 19:492–500. doi: 10.1016/j.concog.2009.10.006 PubMedCrossRefGoogle Scholar
  30. Wright TD, Ward J (2013) The evolution of a visual-to-auditory sensory substitution device using interactive genetic algorithms. Q J Exp Psychol 66:1620–1638CrossRefGoogle Scholar
  31. Zambarbieri D, Schmid R, Magenes G, Prablanc C (1982) Saccadic responses evoked by presentation of visual and auditory targets. Exp Brain Res 47:417–427. doi: 10.1007/BF00239359 PubMedCrossRefGoogle Scholar
  32. Zihl J (1995) Visual scanning behavior in patients with homonymous hemianopia. Neuropsychologia 33:287–303PubMedCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  1. 1.School of Psychology and Sackler Centre for Consciousness ScienceUniversity of SussexBrightonUK

Personalised recommendations