Abstract
In India, there are currently over 18 million blind or visually impaired persons which makes up about 20% of the world’s blind population. The inability to read text or identify objects with ease has impaired their quality of living greatly. Automatic caption generation that describes visual content of images captured in real time has garnered significant attention over the years. Several approaches have been proposed to achieve this task. However, they were not well-suited for mobile devices due to the fact that they were resource-intensive. This paper proposes to employ an attention based LSTM model that performs the task of captioning images that are part of the VizWiz dataset. The generated caption describes the scene captured by the image. The model generates text captions in English and these are converted to voice modality to facilitate the usage by blind people.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Vikram, M., Motwani, V.: Image captioning. arXiv abs/1805.09137 (2018)
Ahsan, H., Bhalla, N., Bhatt, D., Shah, K.: Multi-modal image captioning for the visually impaired. arXiv abs/2105.08106 (2021)
Makav, B., Kılıç, V.: Smartphone-based image captioning for visually and hearing impaired. In: 11th International Conference on Electrical and Electronics Engineering (ELECO), pp. 950–953 (2019). https://doi.org/10.23919/ELECO47770.2019.8990395
Nivedita, M., Asnath Victy Phamila, Y., Harsh, P.V.: Captioning for motion detection for video surveillance applications using deep learning. Int. J. Innov. Technol. Exploring Eng. (IJITEE) 8(8) (2019). ISSN 2278-3075
Mathur, P., Gill, A., Yadav, A., Mishra, A., Bansode, N.K.: Camera2Caption: a real-time image caption generator. In: International Conference on Computational Intelligence in Data Science (ICCIDS), pp. 1–6 (2017). https://doi.org/10.1109/ICCIDS.2017.8272660
Baek, Y., Lee, B., Han, D., Yun, S., Lee, H.: Character region awareness for text detection. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9365–9374 (2019). arXiv:1904.01941
Baek, J., et al.: What is wrong with scene text recognition model comparisons? Dataset and model analysis. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 4714–4722. arXiv:1904.01906
Xu, K., et al.: Show, attend and tell: neural image caption generation with visual attention. In: Proceedings of the 32nd International Conference on Machine Learning, pp. 2048–2057 (2015)
Goyal, P., Pandey, S., Jain, K.: Deep Learning for Natural Language Processing - Creating Neural Networks with Python, pp. 155–156. Apress (2018). ISBN 978-1-4842-3685-7
Papineni, K., Roukos, S., Ward, T., Zhu, W.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), Philadelphia, pp. 311–318 (2002)
Kurama, V.: A Review of Popular Deep Learning Architectures: DenseNet, ResNeXt, MnasNet, and ShuffleNet v2. https://blog.paperspace.com/popular-deep-learning-architectures-densenet-mnasnet-shufflenet/
“Evaluating models”. Google Cloud. https://cloud.google.com/translate/automl/docs/evaluate
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. https://arxiv.org/abs/1810.04805
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Dognin, P., et al.: Image captioning as an assistive technology: lessons learned from VizWiz 2020 challenge. J. Artif. Intell. Res. 73, 437–459 (2022)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 IFIP International Federation for Information Processing
About this paper
Cite this paper
Bhat, A., Kaladharshini, K., Menon, K., Babu, C. (2023). AJNA - Voice Assisted Captioning Tool for the Blind. In: Chandran K R, S., N, S., A, B., Hamead H, S. (eds) Computational Intelligence in Data Science. ICCIDS 2023. IFIP Advances in Information and Communication Technology, vol 673. Springer, Cham. https://doi.org/10.1007/978-3-031-38296-3_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-38296-3_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-38295-6
Online ISBN: 978-3-031-38296-3
eBook Packages: Computer ScienceComputer Science (R0)
