Skip to main content
SpringerLink
Log in

Development of a visual to audio and tactile substitution system for mobility and orientation of visually impaired people: a review

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

According to the World Health Organization (WHO), visual impairment is one of the most common problems affecting approximately one sixth of the world's population. It was found that 39 million of these sufferers have lost their vision completely and require supervision from other people to do their daily chores. Until today, several methodologies have been built to provide effective solutions to visually challenged people, mainly in terms of navigation. The assistive technologies developed for the visually challenged are generally very popular due to the benefits provided, but these technologies are limited in many aspects. One of the main problems with the technologies is that they are non-adaptable and cannot adjust to the changing needs of the participants. The demands of blind people are constantly growing, and there is no powerful technology that can meet all of these demands at the same time. The major constraints with assistive technologies are cost-effectiveness and user-friendliness. This paper presents a complete review of the assistive technologies introduced in the literature to deal with the navigation problems of blind people. Each and every literature work focuses on the requirements of the blind, and several features are considered accordingly. Thus, the presented review provides a detailed description of the evolution of such assistive technologies and the improvements brought in to satisfy the users. Apart from this, the complexities and limitations associated with the technologies are also unwounded to provide a clear perspective on the current scenario for future developers and researchers. Several constraints associated with different assistive technology are interpreted, and this review ends with the insights drawn from the study along with future scopes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23

Similar content being viewed by others

Data availability

Data sharing does not apply to this article.

References

  1. Ang L-M, Seng KP, Heng TZ (2016) Information communication assistive technologies for visually impaired people. Int J Ambient Comput Intell (IJACI) 7(1):45–68

    Google Scholar 

  2. Ashraf MM, Hasan N, Lewis L, Hasan MR, Ray P (2016) A systematic literature review of the application of information communication technology for visually impaired people. Int J Disabil Manag 11:e6

  3. Bai J, Liu Z, Lin Y, Li Y, Lian S, Liu D (2019) Wearable travel aid for environment perception and navigation of visually impaired people. Electronics 8(6):697

    Google Scholar 

  4. Balan O, Moldoveanu A, Moldoveanu F (2015) Navigational audio games: an effective approach toward improving spatial contextual learning for blind people. Int J Disabil Human Dev 14(2):109–118

    Google Scholar 

  5. Bauer Z, Dominguez A, Cruz E, Gomez-Donoso F, Orts-Escolano S, Cazorla M (2020) Enhancing perception for the visually impaired with deep learning techniques and low-cost wearable sensors. Pattern Recogn Lett 137:27–36

    ADS  Google Scholar 

  6. Bhat PG, Rout DK, Subudhi BN, Veerakumar T (2017) Vision sensory substitution to aid the blind in reading and object recognition. In 2017 Fourth International Conference on Image Information Processing (ICIIP), IEEE 1–6

  7. Bouteraa Y (2021) Design and Development of a Wearable Assistive Device Integrating a Fuzzy Decision Support System for Blind and Visually Impaired People. Micromachines 12(9):1082

    PubMed  PubMed Central  Google Scholar 

  8. Brock AM, Truillet P, Oriola B, Picard D, Jouffrais C (2015) Interactivity improves usability of geographic maps for visually impaired people. Human-Computer Interaction 30(2):156–194

    Google Scholar 

  9. Brown DJ, Proulx MJ (2016) Audio–vision substitution for blind individuals: Addressing human information processing capacity limitations. IEEE J Sel Topics Signal Process 10(5):924–931

    ADS  Google Scholar 

  10. Calabrese B, Velázquez R, Del-Valle-Soto C, de Fazio R, Giannoccaro NI, Visconti P (2020) Solar-Powered Deep Learning-Based Recognition System of Daily Used Objects and Human Faces for Assistance of the Visually Impaired. Energies 13(22):6104

    CAS  Google Scholar 

  11. Campisi T, Ignaccolo M, Inturri G, Tesoriere G, Torrisi V (2021) Evaluation of walkability and mobility requirements of visually impaired people in urban spaces. Res Transp Bus Manag 40:100592

    Google Scholar 

  12. Caraiman S, Zvoristeanu O, Burlacu A, Herghelegiu P (2019) Stereo vision based sensory substitution for the visually impaired. Sensors 19(12):2771

    PubMed  PubMed Central  ADS  Google Scholar 

  13. Castle CL, Engward H, Kersey T (2021) Arts activity and well-being for visually impaired military veterans: a narrative discussion of current knowledge. Public Health 194:232–237

    CAS  PubMed  Google Scholar 

  14. Chang W-J, Chen L-B, Hsu C-H, Chen J-H, Yang T-C, Lin C-P (2020) MedGlasses: a wearable smart-glasses-based drug pill recognition system using deep learning for visually impaired chronic patients. IEEE Access 8:17013–17024

    Google Scholar 

  15. Chen Q, Khan M, Tsangouri C, Yang C, Li B, Xiao J and Zhu Z (2017) CCNY smart cane. In 2017 IEEE 7th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER), 1246–1251

  16. Cheng R, Hu W, Chen H, Fang Y, Wang K, Xu Z, Bai J (2021) Hierarchical visual localization for visually impaired people using multimodal images. Expert Syst Appl 165:113743

    Google Scholar 

  17. Cmar JL (2015) Orientation and mobility skills and outcome expectations as predictors of employment for young adults with visual impairments. J Vis Impair Blindness 109(2):95–106

    Google Scholar 

  18. Csapó Á, Wersényi G, Nagy H, Stockman T (2015) A survey of assistive technologies and applications for blind users on mobile platforms: a review and foundation for research. J Multimodal User Interfaces 9(4):275–286

    Google Scholar 

  19. Daudpota MH, Sahito AA, Soomro AM and Channar FS (2017) Giving blind a smart eye: designing and modeling of intelligent white cane for blind people. In 2017 IEEE Global Humanitarian Technology Conference (GHTC), 1–6

  20. Elgendy M, Guzsvinecz T, Sik-Lanyi C (2019) Identification of markers in challenging conditions for people with visual impairment using convolutional neural network. Appl Sci 9(23):5110

    Google Scholar 

  21. Elgendy M, Herperger M, Guzsvinecz T, Lanyi CS (2019) Indoor navigation for people with visual impairment using augmented reality markers. In 2019 10th IEEE International Conference on Cognitive Infocommunications (CogInfoCom), 425–430

  22. Elgendy M, Sik-Lanyi C, Kelemen A (2021) A novel marker detection system for people with visual impairment using the improved tiny-yolov3 model. Comput Methods Programs Biomed 205:106112

    PubMed  Google Scholar 

  23. Elmannai W, Elleithy K (2017) Sensor-based assistive devices for visually-impaired people: current status, challenges, and future directions. Sensors 17(3):565

    PubMed  PubMed Central  ADS  Google Scholar 

  24. Fernandes H, Costa P, Filipe V, Paredes H, Barroso J (2019) A review of assistive spatial orientation and navigation technologies for the visually impaired. Univ Access Inf Soc 18:155–168

    Google Scholar 

  25. Fusco G, Coughlan JM (2020) Indoor localization for visually impaired travelers using computer vision on a smartphone. In Proc 17th Int Web All Conf 1–11 

  26. Gori M, Cappagli G, Tonelli A, Baud-Bovy G, Finocchietti S (2016) Devices for visually impaired people: High technological devices with low user acceptance and no adaptability for children. Neurosci Biobehav Rev 69:79–88

    PubMed  Google Scholar 

  27. Götzelmann T (2018) Visually augmented audio-tactile graphics for visually impaired people. ACM Trans Access Comput (TACCESS) 11(2):1–31

    Google Scholar 

  28. Hamledari H, McCabe B, Davari S (2017) Automated computer vision-based detection of components of under-construction indoor partitions. Autom Constr 74:78–94

    Google Scholar 

  29. Hara K, Azenkot S, Campbell M, Bennett CL, Le V, Pannella S, Moore R, Minckler K, Ng RH, Froehlich JE (2015) Improving public transit accessibility for blind riders by crowdsourcing bus stop landmark locations with google street view: An extended analysis. ACM Trans Accessible Comput (TACCESS) 6(2):1–23

    Google Scholar 

  30. HIMS International. BrailleSense Polaris and U2 (2018) Available online: http://himsintl.com/blindness/ . Accessed on 18 Oct 2018

  31. Hsieh I, Cheng H-C, Ke H-H, Chen H-C, Wang W-J (2021) A CNN-Based Wearable Assistive System for Visually Impaired People Walking Outdoors. Appl Sci 11(21):10026

    CAS  Google Scholar 

  32. Hsieh Y-Z, Lin S-S, Xu F-X (2020) Development of a wearable guide device based on convolutional neural network for blind or visually impaired persons. Multimed Tools Appl 79(39):29473–29491

    Google Scholar 

  33. Hu F, Tang H, Tsema A, Zhu Z (2018) Computer vision for sight: computer vision techniques to assist visually impaired people to navigate in an indoor environment. In: Computer Vision for Assistive Healthcare. Academic Press, pp 1–49

  34. Hussain MA, Ullah MG, Fareed A, Sohail B (2016) The smartcane for blind people an electronically smart stick to aid mobility. Int J Comput Sci Inf Secur 14(4):276

    Google Scholar 

  35. Islam MI, Raj MMH, Nath S, Rahman MF, Hossen S, Imam MH (2018) An indoor navigation system for visually impaired people using a path finding algorithm and a wearable cap. In 2018 3rd International Conference for Convergence in Technology (I2CT), IEEE, 1–6

  36. Islam MM, Sadi MS, Zamli KZ, Ahmed MM (2019) Developing walking assistants for visually impaired people: A review. IEEE Sens J 19(8):2814–2828

    ADS  Google Scholar 

  37. Jacko VA, Choi JH, Carballo A, Charlson B, Moore JE (2015) A new synthesis of sound and tactile music code instruction in a pilot online braille music curriculum. J Vis Impair Blindness 109(2):153–157

    Google Scholar 

  38. Kardyś P, Dabrowski A, Iwanowski M, Huderek D (2016) A new Android application for blind and visually impaired people. In 2016 Signal Processing: Algorithms, Architectures, Arrangements, and Applications (SPA), IEEE, 152–155

  39. Kim M, Park Y, Moon K, Jeong CY (2021) Analysis and Validation of Cross-Modal Generative Adversarial Network for Sensory Substitution. Int J Environ Res Public Health 18(12):6216

    PubMed  PubMed Central  Google Scholar 

  40. Kim S, Park E-S, Ryu E-S (2019) Multimedia vision for the visually impaired through 2d multiarray braille display. Appl Sci 9(5):878

    Google Scholar 

  41. Kim S, Ryu Y, Cho J, Ryu E-S (2019) Towards Tangible Vision for the Visually Impaired through 2D Multiarray Braille Display. Sensors 19(23):5319

    PubMed  PubMed Central  ADS  Google Scholar 

  42. Lahav O, Schloerb DW, Srinivasan MA (2015) Rehabilitation program integrating virtual environment to improve orientation and mobility skills for people who are blind. Comput Educ 80:1–14

    PubMed  PubMed Central  Google Scholar 

  43. Lang L, Gazcón NF, Larrea ML (2018) An open source solution for money bill recognition for the visually impaired user using smartphones. In XXIV Congreso Argentino de Ciencias de la Computación (La Plata, 2018)

  44. Leo M, Medioni G, Trivedi M, Kanade T, Farinella GM (2017) Computer vision for assistive technologies. Comput Vis Image Underst 154:1–15

    Google Scholar 

  45. Li B, Munoz JP, Rong X, Xiao J, Tian Y, Arditi A (2016) ISANA: wearable context-aware indoor assistive navigation with obstacle avoidance for the blind. European Conference on Computer Vision. Springer, Cham, pp 448–462

    Google Scholar 

  46. Mahendran JK, Barry DT, Nivedha AK, Bhandarkar SM (2021) Computer visionbased assistance system for the visually impaired using mobile edge artificial intelligence. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2418–2427

  47. Malik S, Manaf UKA, Ahmad NA, Ismail M (2018) Orientation and Mobility Training in Special Education Curriculum for Social Adjustment Problems of Visually Impaired Children in Pakistan. Int J Instr 11(2):185–202

    Google Scholar 

  48. Mattoccia S, Macrı P (2015) 3D glasses as mobility aid for visually impaired people. In Computer Vision-ECCV 2014 Workshops: Zurich, Switzerland, September 6–7 and 12, 2014, Proceedings, Part III 13, Springer International Publishing, 539–554

  49. Meshram VV, Patil K, Meshram VA, Shu FC (2019) An astute assistive device for mobility and object recognition for visually impaired people. IEEE Trans Human-Mach Syst 49(5):449–460

    Google Scholar 

  50. Moldoveanu A, Balan O, Dascalu M-I, Stanica I, Bodea C-N, Unntorson R, Moldoveanu F (2018) Sound of Vision 3d Virtual Training Environments–A Gamification Approach for Visual to Audio-Haptic Sensory Substitution. Rev Roum Sci Techn-Électrotechn Et Énerg 63(1):112–117

    Google Scholar 

  51. Pawluk DTV, Adams RJ, Kitada R (2015) Designing haptic assistive technology for individuals who are blind or visually impaired. IEEE Trans Haptics 8(3):258–278

    PubMed  Google Scholar 

  52. Pham H-H, Le T-L, Vuillerme N (2016) Real-time obstacle detection system in indoor environment for the visually impaired using microsoft kinect sensor. Journal of Sensors 2016

  53. Pissaloux EE, Velazquez R, Maingreaud F (2017) A new framework for cognitive mobility of visually impaired users in using tactile device. IEEE Trans Human-Mach Syst 47(6):1040–1051

    Google Scholar 

  54. Prescher D, Bornschein J, Köhlmann W, Weber G (2018) Touching graphical applications: bimanual tactile interaction on the HyperBraille pin-matrix display. Univ Access Inf Soc 17(2):391–409

    Google Scholar 

  55. Proulx MJ, Brown DJ, Lloyd-Esenkaya T, Leveson JB, Todorov OS, Watson SH, de Sousa AA (2020) Visual-to-auditory sensory substitution alters language asymmetry in both sighted novices and experienced visually impaired users. Appl Ergon 85:103072

    PubMed  Google Scholar 

  56. Ramadhan AJ (2018) Wearable smart system for visually impaired people. Sensors 18(3):843

    PubMed  PubMed Central  ADS  Google Scholar 

  57. Real S, Araujo A (2019) Navigation systems for the blind and visually impaired: Past work, challenges, and open problems. Sensors 19(15):3404

    PubMed  PubMed Central  ADS  Google Scholar 

  58. Sagale U, Bhutkar G, Karad M, Jathar N (2018) An Eye-Free Android application for visually impaired users. Ergonomics in Caring for People. Springer, Singapore, pp 291–297

    Google Scholar 

  59. Sanches P, Christie E, Bueno J, Okimoto MLLR (2021) Designing 3D Printed Audio-Tactile Graphics: Recommendations from Prior Research. International Conference on Human-Computer Interaction. Springer, Cham, pp 461–472

    Google Scholar 

  60. Semary NA, Fadl SM, Essa MS, Gad AF (2015) Currency recognition system for visually impaired: Egyptian banknote as a study case. In 2015 5th International Conference on Information & Communication Technology and Accessibility (ICTA), IEEE, 1–6

  61. Shaikh F, Meghani MA, Kuvar V, Pappu S (2018) Wearable navigation and assistive system for visually impaired. In 2018 2nd International Conference on Trends in Electronics and Informatics (ICOEI), IEEE, 747–751

  62. Shultz C, Peshkin M, Colgate JE (2018) The application of tactile, audible, and ultrasonic forces to human fingertips using broadband electroadhesion. IEEE Trans Haptics 11(2):279–290

    PubMed  Google Scholar 

  63. Simões WCSS, Machado GS, Sales A, de Lucena MM, Jazdi N, de Lucena VF (2020) A review of technologies and techniques for indoor navigation systems for the visually impaired. Sensors 20(14):3935

    PubMed  PubMed Central  ADS  Google Scholar 

  64. Sivan S and Darsan G (2016) Computer vision based assistive technology for blind and visually impaired people. In Proceedings of the 7th International Conference on Computing Communication and Networking Technologies, 1–8

  65. Sufri NAJ, Rahmad NA, As’ari MA, Zakaria NA, Jamaludin MN, Ismail LH, Mahmood NH (2017) Image based ringgit banknote recognition for visually impaired. J Telecommun Electron Comput Eng (JTEC) 9(3–9):103–111

    Google Scholar 

  66. Tan CJ, Wong WK, Min TS (2020) Malaysian Banknote Reader for Visually Impaired Person. In 2020 IEEE Student Conference on Research and Development (SCOReD), 74–79

  67. Tapu R, Mocanu B, Zaharia T (2017) A computer vision-based perception system for visually impaired. Multimed Tools Appl 76:11771–11807

    Google Scholar 

  68. Tseng Y-C, Liu SH-Y, Lou M-F, Huang G-S (2018) Quality of life in older adults with sensory impairments: a systematic review. Qual Life Res 27:1957–1971

    PubMed  Google Scholar 

  69. Tsirmpas C, Rompas A, Fokou O, Koutsouris D (2015) An indoor navigation system for visually impaired and elderly people based on Radio Frequency Identification (RFID). Inf Sci 320:288–305

    Google Scholar 

  70. Uematsu H, Suzuki M, Kanno Y and Kajimoto H (2016) Tactile vision substitution with tablet and electro-tactile display. In Haptics: Perception, Devices, Control, and Applications: 10th International Conference, EuroHaptics 2016, London, UK, July 4–7, 2016, Proceedings, Part I 10, Springer International Publishing, 503–511

  71. Xiao J, Joseph SL, Zhang X, Li B, Li X, Zhang J (2015) An assistive navigation framework for the visually impaired. IEEE Trans Human-Mach Syst 45(5):635–640

    Google Scholar 

  72. Zhang X, Yao X, Zhu Y, Hu F (2019) An ARCore based user-centric assistive navigation system for visually impaired people. Appl Sci 9(5):989

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Technology DOT, SPPU & PICT, Pune, Maharashtra, 411007, India

    Komal Mahadeo Masal

  2. Vishwakarma Institute of Technology, Pune, Maharashtra, 411037, India

    Shripad Bhatlawande

  3. Department of Technology DOT, SPPU & MIT Academy of Engineering, Pune, Maharashtra, 411007, India

    Sachin Dattatraya Shingade

Authors
  1. Komal Mahadeo Masal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shripad Bhatlawande
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sachin Dattatraya Shingade
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors have equal contributions to this work.

Corresponding author

Correspondence to Komal Mahadeo Masal.

Ethics declarations

Ethical Approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Consent to participate

All the authors involved have agreed to participate in this submitted article.

Consent for Publication

All the authors involved in this manuscript give full consent to publish this submitted article.

Conflict of Interest

Authors declare that they have no conflict of interest.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Masal, K.M., Bhatlawande, S. & Shingade, S.D. Development of a visual to audio and tactile substitution system for mobility and orientation of visually impaired people: a review. Multimed Tools Appl 83, 20387–20427 (2024). https://doi.org/10.1007/s11042-023-16355-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16355-0

Keywords

Search

Navigation