Abstract
A VQA system takes a picture and an open-ended query in natural language related to the image as inputs and outputs a response in natural language. In this paper, we aim to comprehend numerous methods devised by researchers and compare their performance on various datasets. This includes several methods including Bilinear models, Attention and Non-Attention models, Multimodal approach, etc. Additionally, we have proposed a Hybrid Co-Attention model that addresses visual and linguistic features simultaneously over different levels to find semantic overlaps between them. We were able to get an accuracy that is similar to state-of-the-art models by altering only the text-level features, i.e., training accuracy of 57.02% and validation accuracy of 42.78%. We also got top 5 accuracy of 93.47% on the training set and 77.67% on the validation dataset.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Lu J, Yang J, Batra D, Parikh D (2016) Hierarchical question-image co-attention for visual question answering. Advances in neural information processing systems, p 29
Guo D, Xu C, Tao D (2021) Bilinear graph networks for visual question answering. IEEE Trans Neural Netw Learn Syst
Ben-Younes H, Cadene R, Cord M, Thome N (2017) Mutan: multimodal tucker fusion for visual question answering. In: Proceedings of the IEEE international conference on computer vision, pp 2612–2620
Chen YC, Li L, Yu L, El Kholy A, Ahmed F, Gan Z, Cheng Y, Liu J (2020) Uniter: universal image-text representation learning. In: European conference on computer vision. Springer, Cham, pp 104–120
Nguyen DK, Goswami V, Chen X (2020) Movie: revisiting modulated convolutions for visual counting and beyond. arXiv preprint arXiv:2004.11883
Anderson P, He X, Buehler C, Teney D, Johnson M, Gould S, Zhang L (2018) Bottom-up and top-down attention for image captioning and visual question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6077–6086
Li X, Yin X, Li C, Zhang P, Hu X, Zhang L, Wang L, Hu H, Dong L, Wei F, Choi Y (2020) Oscar: object-semantics aligned pre-training for vision-language tasks. In: European conference on computer vision. Springer, Cham, pp 121–137
Luo M, Zeng Y, Banerjee P, Baral C (2021) Weakly-supervised visual-retriever-reader for knowledge-based question answering. arXiv preprint arXiv:2109.04014
Su W, Zhu X, Cao Y, Li B, Lu L, Wei F, Dai J (2019) Vl-Bert: pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530
Agrawal A, Batra D, Parikh Dp (2016) Analyzing the behavior of visual question answering models. arXiv preprint arXiv:1606.07356
Yu Z, Yu J, Cui Y, Tao D, Tian Q (2019) Deep modular co-attention networks for visual question answering. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 6281–6290
Han X, Wang S, Su C, Huang Q, Tian Q (2021) Greedy gradient ensemble for robust visual question answering. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 1584–1593
Hu R, Andreas J, Rohrbach M, Darrell T, Saenko K (2017) Learning to reason: end-to-end module networks for visual question answering. In: Proceedings of the IEEE international conference on computer vision, pp 804–813
Andreas J, Rohrbach M, Darrell T, Klein D (2016) Deep compositional question answering with neural module networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 39–48
Gamage BMSV, Hong LC (2021) Improved RAMEN: towards domain generalization for visual question answering. arXiv preprint arXiv:1903.00366
Singh A, Natarajan V, Shah M, Jiang Y, Chen X, Batra D, Parikh D, Rohrbach M (2019) Towards VQA models that can read. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8317–8326
Wang T, Huang J, Zhang H, Sun Q (2020) Visual commonsense R-CNN. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10760–10770
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Arora, M., Mudgil, P., Kumar, R., Kapoor, T., Gupta, R., Agnihotri, A. (2023). A Novel Approach to Visual Linguistics by Assessing Multi-level Language Substructures. In: Khanna, A., Polkowski, Z., Castillo, O. (eds) Proceedings of Data Analytics and Management . Lecture Notes in Networks and Systems, vol 572. Springer, Singapore. https://doi.org/10.1007/978-981-19-7615-5_12
Download citation
DOI: https://doi.org/10.1007/978-981-19-7615-5_12
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-19-7614-8
Online ISBN: 978-981-19-7615-5
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)