Sound of Vision – 3D Scene Reconstruction from Stereo Vision in an Electronic Travel Aid for the Visually Impaired

  • Mateusz Owczarek
  • Piotr Skulimowski
  • Pawel Strumillo
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9759)


The paper presents the preliminary results for the parametrization of 3D scene for sonification purposes in an electronic travel aid (ETA) system being built within the European Union’s H2020 Sound of Vision project. The ETA is based on the concept of sensory substitution, in which visual information is transformed into either acoustic or haptic stimuli. In this communication we concentrate on vision-to-audio conversion i.e. employing stereovision for reconstruction of 3D scenes and building a spatial model of the environment for sonification. Two prerequisite approaches for the sonification are proposed. One involves the direct sonification of the so-called “U-disparity” representation of the depth map of the environment, while the other relies on the processing of the depth map to extract obstacles present in the environment and presenting them to the user as auditory icons reflecting specific size and location of the sonified object.


Electronic Travel Aid Stereo vision UV-Disparity Ground plane detection Obstacle detection 



This work received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 643636 “Sound of Vision”.


  1. 1.
    Hersh, M., Johnson, M.: Assistive Technology for Visually Impaired and Blind People. Springer, London (2008)CrossRefGoogle Scholar
  2. 2.
    Ivanchenko, V., Coughlan J., Huiying, S.: Detecting and locating crosswalks using a camera phone. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop, CVPRW 2008, pp. 1–8, 23–28 June 2008Google Scholar
  3. 3.
    Huiying, S., Coughlan, J.: Reading LCD/LED displays with a camera cell phone. In: IEEE Conference on Computer Vision and Pattern Recognition Workshop, CVPRW 2006, p. 119, 17–22 June 2006Google Scholar
  4. 4.
    Bach-y-Rita, P.: Brain Mechanisms in Sensory Substitution. Academic Press, New York (1972)Google Scholar
  5. 5.
    Strumiłło, P.: Electronic personal navigation systems for the blind and visually impaired (in Polish), Faculty of Electrical, Electronic, Computer and Control Engineering, Lodz University of Technology, Lodz (2012)Google Scholar
  6. 6.
    Sound of Vision: Natural sense of vision through acoustics and haptics. . Accessed 25 Mar 2016
  7. 7.
    Flores, G., Kurniawan, S., Manduchi, R., Martinson, E., Morales, L.M., Sisbot, E.A.: Vibrotactile guidance for wayfinding of blind walkers. IEEE Trans. Haptics 8(3), 306–317 (2015)CrossRefGoogle Scholar
  8. 8.
    Dakopoulos, D., Bourbakis, N.G.: Wearable obstacle avoidance electronic travel aids for blind: a Survey. IEEE Trans. Syst. Man Cybern. – Part C: Appl. Rev. 40(1), 25–35 (2010)CrossRefGoogle Scholar
  9. 9.
    Garaj, V., Jirawimut, R., Ptasinski, P., Cecelja, F., Balachandran, W.: A system for remote sighted guidance of visually impaired pedestrians. Br. J. Vis. Impairment 21, 55–63 (2003)CrossRefGoogle Scholar
  10. 10.
    Barański, P., Strumiłło, P.: Emphatic trials of a teleassistance system for the visually impaired. J. Med. Imaging Health Inf. 5(8), 1640–1651 (2015)CrossRefGoogle Scholar
  11. 11.
    vOICe: . Accessed 25 Mar 2016
  12. 12.
    Balakrishnan, G., Sainarayanan, G., Nagarajan, R., Sazali, Y.: A stereo image processing system for visually impaired. Int. J. Signal Process. 2(3), 136–145 (2006)Google Scholar
  13. 13.
    González-Mora, J., Rodríguez-Hernández, A., Rodríguez-Ramos, L., Díaz-Saco, L., Sosa, N.: Development of a new space perception system for blind people, based on the creation of a virtual acoustic space. In: Mira, J. (ed.) Engineering Applications of Bio-Inspired Artificial Neural Networks. LNCS, vol. 1607, pp. 321–330. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  14. 14.
    Heyes, D.: The Sonic pathfinder: a new electronic travel aid. J. Vis. Impairment Blindness 77, 200–202 (1984)Google Scholar
  15. 15.
    Bujacz, M., Skulimowski, P., Strumillo, P.: Naviton – a prototype mobility aid for auditory presentation of 3D scenes. J. Audio Eng. Soc. 60(9), 696–708 (2012)Google Scholar
  16. 16.
    Chunlei, Y., Cherfaoui, V., Bonnifait, P.: Evidential occupancy grid mapping with stereo-vision. In: IEEE Intelligent Vehicles Symposium (IV), pp. 712–717, 28 June 2015 – 1 July 2015. doi: 10.1109/IVS.2015.7225768
  17. 17.
    Zhencheng, H., Uchimura, K.: U-V-disparity: an efficient algorithm for stereovision based scene analysis. In: IEEE Intelligent Vehicles Symposium, pp. 48–54, 6–8 June 2005. doi: 10.1109/IVS.2005.1505076
  18. 18.
    Labayrade, R., Aubert, D., Tarel, J.-P.: Real time obstacle detection in stereovision on non flat road geometry through “V-disparity” representation. In: IEEE Intelligent Vehicle Symposium, vol. 2, pp. 646–651, 17–21 June 2002. doi: 10.1109/IVS.2002.1188024
  19. 19.
    Zhencheng, H., Lamosa, F., Uchimura, K.: A complete U-V-disparity study for stereovision based 3D driving environment analysis. In: Fifth International Conference on 3-D Digital Imaging and Modeling, pp. 204–211, 13–16 June 2005. doi: 10.1109/3DIM.2005.6
  20. 20.
    Suzuki, S., Abe, K.: Topological structural analysis of digitized binary images by border following. Comput. Vis. Graph. Image Process. 30(1), 32–46 (1985)CrossRefzbMATHGoogle Scholar
  21. 21.
    Bujacz, M., et al.: Sound of vision - spatial audio output and sonification approaches. In: Miesenberger, K., Bühler, C., Penaz, P. (eds.) ICCHP 2016, Part II. LNCS, vol. 9759, pp. 200–207. Springer, Heidelberg (2016) Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Mateusz Owczarek
    • 1
  • Piotr Skulimowski
    • 1
  • Pawel Strumillo
    • 1
  1. 1.Institute of ElectronicsLodz University of TechnologyŁódźPoland

Personalised recommendations