Temporal Convolutional and Recurrent Networks for Image Captioning

  • Natalia IskraEmail author
  • Vitaly Iskra
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 1055)


Recently temporal convolutional networks have shown excellent qualities in sequence modeling tasks [1]. Taking this fact into account, in this paper we investigate the possibilities of replacing recurrent networks in architectures targeted specifically at image captioning. We evaluate the solution on Visual Genome dataset [2], which provides extensive set of labels and descriptions that thoroughly grounds visual concepts to natural language.


Image captioning Convolutional neural networks Recurrent neural networks Visual Genome Dilated convolution Weight normalization Dropout Adam optimization 


  1. 1.
    Bai, S., Colter, J.Z., Coltun, V.: An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 (2018)
  2. 2.
    Krishna, R., Zhu, Y., Groth, O., et al.: Visual genome: connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis. 1(123), 32–73 (2017)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Johnson, J., Karpathy, A., Fei-Fei, L.: DenseCap: fully convolutional localization networks for dense captioning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4565–4574 (2016)Google Scholar
  4. 4.
    Iskra, N., Iskra, V., Lukashevich M.: Neural network based image understanding with ontological approach. In: Open Semantic Technologies for Intelligent Systems (OSTIS-2019), pp. 113–122 (2019)Google Scholar
  5. 5.
    Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). Scholar
  6. 6.
    Kulkarni, G., et al.: Baby talk: understanding and generating image descriptions. In: Proceedings of the 24th Conference on Computer Vision and Pattern Recognition (2011)Google Scholar
  7. 7.
    Sun, C., Gan, C., Nevatia, R.: Automatic concept discovery from parallel text and visual corpora. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2596–2604 (2015)Google Scholar
  8. 8.
    Hossain, M.D., Sohel, F., Shiratuddin, M.F., Laga, H.: A comprehensive survey of deep learning for image captioning. ACM Comput. Surv. (CSUR) 6(51), 118 (2019)Google Scholar
  9. 9.
    Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3128–3137 (2015)Google Scholar
  10. 10.
    Liu, D., Hanwang, Zh., Zheng-Jun, Zh., Feng, W.: Explainability by parsing: neural module tree networks for natural language visual grounding. arXiv preprint, arXiv:1812.03299 (2018)
  11. 11.
    Jiang, W., Ma, L., Jiang, Y.-G., Liu, W., Zhang, T.: Recurrent fusion network for image captioning. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11206, pp. 510–526. Springer, Cham (2018). Scholar
  12. 12.
    Wang, Q., Chan, A.B.: CNN + CNN: Convolutional decoders for image captioning. arXiv preprint, arXiv:1805.09019 (2018)
  13. 13.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)Google Scholar
  14. 14.
    Salimans, T., Kingma, D.: Weight normalization: a simple reparameterization to accelerate training of deep neural networks. In: Advances in Neural Information Processing Systems, pp. 901–909 (2016)Google Scholar
  15. 15.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  16. 16.
    Kingma, D., Ba., J.: Adam: a method for stochastic optimization. arXiv preprint, arXiv:1412.6980 (2014)
  17. 17.
    Banerjee, S., Lavie, A.: METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization (2005)Google Scholar
  18. 18.
    Iskra, N., Shunkevich, D.: Ontological approach to image captioning evaluation. In: Proceedings of Pattern Recognition and Information Processing, pp. 219–223 (2019)Google Scholar
  19. 19.
    Vedantam, R., Zitnick, C.L., Parikh, D.: Cider: consensus-based image description evaluation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4566–4575 (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Belarusian State University of Informatics and RadioelectronicsMinskRepublic of Belarus
  2. 2.Omnigon Communications LLCNew YorkUSA

Personalised recommendations