Skip to main content
SpringerLink
Log in

Experimenting Encoder-Decoder Architecture for Visual Image Captioning

  • Conference paper
  • First Online:
Cognition and Recognition (ICCR 2021)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1697))

Included in the following conference series:

Abstract

Image captioning is an essential task in artificial intelligence that predicts the description of a given input image. In recent years, both computer vision and natural language processing witnessed huge advancement making it capable to extract high-level semantic information and process it to structure new sentences based on that. A myriad of research has been done discussing the use of deep learning in image caption tasks. This paper explores the various combination of CNN and RNN modules in Encoder-decoder architecture to find the best image caption generator. The different models were compared using BLEU and CIDEr metrics. The proposed model with fine-tuned parameters showed a BLEU as high as 67.2, 59.8, 53, 44.7, and CIDEr score was 46 with Restnet50v2 model as an encoder and GRU as a decoder. After a little model hyperparameter tuning on Batch size and learning rate, an improvement of 15% and 12.5% was achieved in CIDEr and BLEU-4score.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Farhadi, A., Hejrati, M., Sadeghi, M.A., Young, P., Rashtchian, C., Hockenmaier, J., Forsyth, D.: Every picture tells a story: generating sentences from images. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6314, pp. 15–29. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15561-1_2

    Chapter  Google Scholar 

  2. Ordonez, V., Kulkarni, G., Berg, T.: Im2text: describing images using 1 million captioned photographs. In: Advances in Neural Information Processing Systems (NIPS), pp. 1143–1151 (2011)

    Google Scholar 

  3. Kuznetsova, P., Ordonez, V., Berg, A.: Collective generation of natural image descriptions. In: Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, 1 July, pp. 359–368 (2012)

    Google Scholar 

  4. Mason, R., Charniak, E.: Nonparametric method for data-driven image captioning. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, Stroudsburg, PA, USA, pp. 592–598. Association for Computational Linguistics (2014)

    Google Scholar 

  5. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  6. Yang, Y., Teo, C., Daumé III, H., Aloimonos, Y.: Corpus-guided sentence generation of natural images. In: Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pp. 444–454 (2011)

    Google Scholar 

  7. Kulkarni, G., et al.: Baby talk: understanding and generating simple image descriptions. In: Proceedings of the 24th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2011), vol. 18, pp. 1601–1608 (2011)

    Google Scholar 

  8. Li, S., Kulkarni, G., Berg, T., Berg, A., Choi, Y.: Composing simple image descriptions using web-scale n-grams. In: Proceedings of the 15th Conference on Computational Natural Language Learning (CoNLL 2011), Portland, USA, pp. 220–228. Association for Computational Linguistics (2011)

    Google Scholar 

  9. Parikh, H., et al.: Encoder-decoder architecture for image caption generation. In: Proceedings of the 3rd International Conference on Communication System, Computing and IT Applications (CSCITA). IEEE (2020)

    Google Scholar 

  10. Chen, H., Lin, H., Yao, M.: Improving the efficiency of encoder-decoder architecture for pixel-level crack detection. IEEE Access (2019). https://doi.org/10.1109/ACCESS.2019.2961375

    Article  Google Scholar 

  11. Wang, Z., Su, X., Ding, Z.: Long-term traffic prediction based on LSTM encoder-decoder architecture. IEEE Trans. Intell. Transp. Syst. (2020). https://doi.org/10.1109/TITS.2020.2995546

    Article  Google Scholar 

  12. Wang, C., Yang, H., Bartz, C., Meinel, C.: Image captioning with deep bidirectional LSTMs. In: Proceedings of the 2016 ACM on Multimedia Conference, pp. 988–997. ACM (2016)

    Google Scholar 

  13. Rennie, S.J., Marcheret, E., Mroueh, Y., Ross, J., Goel, V.: Self-critical sequence training for image captioning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), pp. 1179–1195 (2017)

    Google Scholar 

  14. Aneja, J., Deshpande, A., Schwing, A.G.: Convolutional image captioning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5561–5570 (2018)

    Google Scholar 

  15. Kiros, R., Salakhutdinov, R., Zemel, R.: Multimodal neural language models. In: Proceedings of the 31st International Conference on Machine Learning (ICML), pp. 595–603 (2014)

    Google Scholar 

  16. Yao, T., Pan, Y., Li, Y., Qiu, Z., Mei, T.: Boosting image captioning with attributes. In: IEEE International Conference on Computer Vision (ICCV), pp. 4904–4912 (2017)

    Google Scholar 

  17. Huo, L., Bai, L., Zhou, S.-M.: Automatically generating natural language descriptions of images by a deep hierarchical framework. IEEE Trans. Cybern. 52, 1–12 (2021). https://doi.org/10.1109/TCYB.2020.3041595

    Article  Google Scholar 

  18. Balakrishna, K.: WSN, APSim, and communication model-based irrigation optimization for horticulture crops in real time. In: Tomar, P., Kaur, G. (eds.) Artificial Intelligence and IoT-Based Technologies for Sustainable Farming and Smart Agriculture, pp. 243–254. IGI Global (2021). https://doi.org/10.4018/978-1-7998-1722-2.ch015

  19. Hou, D., Zhao, Z., Liu, Y., Chang, F., Hu, S.: Automatic report generation for chest X-ray images via adversarial reinforcement learning. IEEE Access 9, 21236–21250 (2019). https://doi.org/10.1109/ACCESS.2021.3056175

    Article  Google Scholar 

  20. de Oliveira Junior, L.A., Medeiros, H.R., Macedo, D., Zanchettin, C., Oliveira, A.L.I., Ludermir, T.: SegNetRes-CRF: a deep convolutional encoder-decoder architecture for semantic image segmentation. In: International Joint Conference on Neural Networks (IJCNN) (2018)

    Google Scholar 

  21. https://ai.googleblog.com/2020/06/recent-advances-in-google-translate.html

  22. Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: a neural image caption generator. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3156–3164 (2015)

    Google Scholar 

  23. Dai, B., Fidler, S., Urtasun, R., Lin, D.: Towards diverse and natural image descriptions via a conditional gan. In: ICCV, pp. 2970–2979 (2017)

    Google Scholar 

  24. Dai, B., Lin, D.: Contrastive learning for image captioning. In: Advances in Neural Information Processing Systems, pp. 898–907 (2017)

    Google Scholar 

  25. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (2015)

    Google Scholar 

  26. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)

    Google Scholar 

  27. Szegedy, I.C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Computer Vision and Pattern Recognition (2016). https://doi.org/10.1109/CVPR.2016.308

  28. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  29. Chung, J., Gülçehre, Ç., Cho, K., and Bengio, Y.: Empirical evaluation of gated recurrent neural networks on sequence modeling. In: NIPS 2014 Deep Learning and Representation Learning Workshop (2014)

    Google Scholar 

  30. Rashtchian, C., Young, P., Hodosh, M., Hockenmaier, J.: Collecting image annotations using Amazon’s mechanical turk. In: NAACL HLT Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, pp. 139–147 (2010)

    Google Scholar 

  31. Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, p. 311 (2002)

    Google Scholar 

  32. Vedantam, R., Zitnick, C.L., Parikh, D.: CIDEr: consensus-based image description evaluation. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4566–4575 (2015)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. State University of New York Buffalo, Buffalo, USA

    Hasan Asif

Authors
  1. Hasan Asif
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hasan Asif .

Editor information

Editors and Affiliations

  1. University of Mysore, Mysore, India

    D. S. Guru

  2. Maharaja Institute of Technology Mysore, Mandya, India

    Sharath Kumar Y. H.

  3. Maharaja Institute of Technology Mysore, Mandya, India

    Balakrishna K.

  4. Jawaharlal Nehru University, New Delhi, India

    R. K. Agrawal

  5. Tokyo Denki University, Tokyo, Japan

    Manabu Ichino

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Asif, H. (2022). Experimenting Encoder-Decoder Architecture for Visual Image Captioning. In: Guru, D.S., Y. H., S.K., K., B., Agrawal, R.K., Ichino, M. (eds) Cognition and Recognition. ICCR 2021. Communications in Computer and Information Science, vol 1697. Springer, Cham. https://doi.org/10.1007/978-3-031-22405-8_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-22405-8_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-22404-1

  • Online ISBN: 978-3-031-22405-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation