Skip to main content
SpringerLink
Log in

Improved Bengali Image Captioning via Deep Convolutional Neural Network Based Encoder-Decoder Model

  • Conference paper
  • First Online:
Proceedings of International Joint Conference on Advances in Computational Intelligence

Part of the book series: Algorithms for Intelligent Systems ((AIS))

Abstract

Image Captioning is an arduous task of producing syntactically and semantically correct textual descriptions of an image in natural language with context related to the image. Existing notable pieces of research in Bengali Image Captioning (BIC) are based on encoder-decoder architecture. This paper presents an end-to-end image captioning system utilizing a multimodal architecture by combining a one-dimensional convolutional neural network (CNN) to encode sequence information with a pre-trained ResNet-50 model image encoder for extracting region-based visual features. We investigate our approach’s performance on the BanglaLekhaImageCaptions dataset using the existing evaluation metrics and perform a human evaluation for qualitative analysis. Experiments show that our approach’s language encoder captures the fine-grained information in the caption, and combined with the image features, it generates accurate and diversified caption. Our work outperforms all the existing BIC works and achieves a new state-of-the-art (SOTA) performance by scoring 0.651 on BLUE-1, 0.572 on CIDEr, 0.297 on METEOR, 0.434 on ROUGE, and 0.357 on SPICE.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/FaiyazKhan11/Improved-Bengali-Image-Captioning-via-deep-convolutional-neural-network-based-encoder-decoder-model.

  2. 2.

    https://github.com/salaniz/pycocoevalcap.

  3. 3.

    https://github.com.

References

  1. Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. In: Advances in neural information processing systems, pp 3104–3112

    Google Scholar 

  2. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  3. Rahman M, Mohammed N, Mansoor N, Momen S (2019) Chittron: an automatic Bangla image captioning system. Procedia Computer Sci 154:636–642

    Article  Google Scholar 

  4. Deb T, Ali MZA, Bhowmik S, Firoze A, Ahmed SS, Tahmeed MA, Rah-man N, Rahman RM (2019) Oboyob: a sequential-semantic bengali image captioning engine. J Intell Fuzzy Syst (Preprint) 1–13

    Google Scholar 

  5. Tanti M, Gatt A, Camilleri K (2017) What is the role of recurrent neural networks (RNNs) in an image caption generator? In: Proceedings of the 10th international conference on natural language generation. Association for Computational Linguistics, Santiago de Compostela, Spain, pp 51–60

    Google Scholar 

  6. Collobert R, Weston J, Bottou L, Karlen M, Kavukcuoglu K, Kuksa P (2011) Natural language processing (almost) from scratch. J Mach Learn Res 12:2493–2537

    MATH  Google Scholar 

  7. Wang Q, Chan AB (2018) CNN + CNN: Convolutional decoders for image captioning. In:31st IEEE/CVF conference on computer vision and pattern recognition (CVPR2018)

    Google Scholar 

  8. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

    Google Scholar 

  9. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) ImageNet: a large-scale hierarchical image database. In: CVPR09

    Google Scholar 

  10. Mansoor NK, Mohammed AH, Momen N, Rahman S, Matiur M (2019) Banglalekhaimagecaptions, mendeleydata. https://doi.org/10.17632/rxxch9vw59.2

  11. Gerber R, Nagel NH (1996) Knowledge representation for the generation of quantified natural language descriptions of vehicle traffic in image sequences. In: Proceedings of 3rd IEEE international conference on image processing, vol 2. IEEE, pp 805–808

    Google Scholar 

  12. Duygulu P, Barnard K, de Freitas JF, Forsyth DA (2002) Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary. European conference on computer vision. Springer, Berlin, pp 97–112

    Google Scholar 

  13. Bahdanau D, Cho K, Bengio Y (2015) Neural machine translation by jointly learning to align and translate. In: 3rd international conference on learning representations (ICLR 2015)

    Google Scholar 

  14. Vinyals O, Toshev A, Bengio S, Erhan D (2015) Show and tell: a neural image caption generator. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3156–3164

    Google Scholar 

  15. Donahue J, Anne Hendricks L, Guadarrama S, Rohrbach M, Venugopalan S, Saenko K, Darrell T (2015) Long-term recurrent convolutional networks for visual recognition and description. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2625–2634

    Google Scholar 

  16. Johnson J, Karpathy A, Fei-Fei L (2016) Densecap: fully convolutional localization networks for dense captioning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4565–4574

    Google Scholar 

  17. Xu K, Ba J, Kiros R, Cho K, Courville A, Salakhudinov R, Zemel R, Bengio Y (2015) Show, attend and tell: neural image caption generation with visual attention. In: International conference on machine learning, pp 2048–2057

    Google Scholar 

  18. You Q, Jin H, Wang Z, Fang C, Luo J (2016) Image captioning with semantic attention. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4651–4659

    Google Scholar 

  19. Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft coco: common objects in context. In: European conference on computer vision. Springer, Berlin, pp 740–755

    Google Scholar 

  20. Young P, Lai A, Hodosh M, Hockenmaier J (2014) From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Trans Assoc Comput Linguistics 2:67–78

    Article  Google Scholar 

  21. Hodosh M, Young P, Hockenmaier J (2013) Framing image description as a ranking task: data, models and evaluation metrics. J Artif Intell Res 47:853–899

    Article  MathSciNet  Google Scholar 

  22. Yoshikawa Y, Shigeto Y, Takeuchi A (2017) Stair captions: constructing a large-scale japanese image caption dataset. In: Proceedings of the 55th annual meeting of the Association for Computational Linguistics (vol 2: short papers), pp 417–421

    Google Scholar 

  23. Li X, Lan W, Dong J, Liu H (2016) Adding chinese captions to images. In: Proceedings of the 2016 ACM on international conference on multimedia retrieval, pp 271–275

    Google Scholar 

  24. Elliott D, Frank S, Sima’an K, Specia L (2016) Multi30k: multilingual English-German image descriptions. In: Proceedings of the 5th workshop on vision and language, pp 70–74

    Google Scholar 

  25. Al-Muzaini HA, Al-Yahya TN, Benhidour H (2018) Automatic arabic image captioning using RNN-LSTM-based language model and CNN. Int J Adv Comput Sci Appl 9(6)

    Google Scholar 

  26. Papineni K, Roukos S, Ward T, Zhu WJ (2002) Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp 311–318

    Google Scholar 

  27. Denkowski M, Lavie A (2014) Meteor universal: language specific translation evaluation for any target language. In: Proceedings of the ninth workshop on statistical machine translation, pp 376–380

    Google Scholar 

  28. Lin CY (2004) Rouge: a package for automatic evaluation of summaries. In: Text summarization branches out, pp 74–81

    Google Scholar 

  29. Vedantam R, Lawrence Zitnick C, Parikh D (2015) Cider: consensus-based image description evaluation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4566–4575

    Google Scholar 

  30. Anderson P, Fernando B, Johnson M, Gould S (2016) Spice: semantic propositional image caption evaluation. European conference on computer vision. Springer, Berlin, pp 382–398

    Google Scholar 

  31. Chen X, Fang H, Lin TY, Vedantam R, Gupta S, Dollár P, Zitnick CL (2015) Microsoft coco captions: data collection and evaluation server. arXiv preprint arXiv:1504.00325

  32. Devlin J, Chang MW, Lee K, Toutanova K (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the Association for Computational Linguistics: Human Language Technologies, vol 1 (Long and Short Papers), pp 4171–4186

    Google Scholar 

  33. Yang Z, Dai Z, Yang Y, Carbonell J, Salakhutdinov RR, Le QV (2019) Xlnet: generalized auto regressive pretraining for language understanding. In: Advances in neural information processing systems, pp 5753–5763

    Google Scholar 

  34. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  35. Conneau A, Lample G (2019) Cross-lingual language model pretraining. In: Advances in neural information processing systems, pp 7059–7069

    Google Scholar 

Download references

Acknowledgements

We want to thank the Natural Language Processing Group, Dept. of CSE, SUST, for their valuable guidelines in our research work.

Author information

Authors and Affiliations

  1. Dept of Computer Science & Engineering, Shahjalal University of Science and Technology, Sylhet, Bangladesh

    Mohammad Faiyaz Khan, S. M. Sadiq-Ur-Rahman & Md. Saiful Islam

Authors
  1. Mohammad Faiyaz Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S. M. Sadiq-Ur-Rahman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Md. Saiful Islam
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Md. Saiful Islam .

Editor information

Editors and Affiliations

  1. Department of Computer Science and Engineering, Jahangirnagar University, Dhaka, Bangladesh

    Mohammad Shorif Uddin

  2. Department of Applied Mathematics, South Asian University, New Delhi, India

    Jagdish Chand Bansal

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Faiyaz Khan, M., Sadiq-Ur-Rahman, S.M., Saiful Islam, M. (2021). Improved Bengali Image Captioning via Deep Convolutional Neural Network Based Encoder-Decoder Model. In: Uddin, M.S., Bansal, J.C. (eds) Proceedings of International Joint Conference on Advances in Computational Intelligence. Algorithms for Intelligent Systems. Springer, Singapore. https://doi.org/10.1007/978-981-16-0586-4_18

Download citation

Publish with us

Policies and ethics

Search

Navigation