Advertisement

Adaptive Joint Attention with Reinforcement Training for Convolutional Image Caption

  • Ruoyu Chen
  • Zhongnian Li
  • Daoqiang ZhangEmail author
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 1072)

Abstract

A convolutional decoder for image caption has proven to be easier to train than the Long Short Term Memory (LSTM) decoder [2]. However, previous convolutional image captioning methods are not good at capture the relationship between generated words, which could lead to failure in predicting visually unrelated words. To address this issue, we propose an Adaptive Joint Attention (AJA) module for the convolutional decoder, by incorporating self-attention to explore the correlation between generated words automatically. Specifically, we develop a word gate with multi-head self-attention to directly capture words’ relationship. An adaptive combination strategy is proposed to learn which word is less related to image information. Besides, reinforcement learning is applied for directly optimizing the non-differentiable metrics while avoiding the exposure bias during inference. Extensive evaluations on two public data sets demonstrate that our method outperforms several state-of-the-art approaches in image captioning task.

Keywords

Computer vision Image caption Nature language generation 

References

  1. 1.
    Anderson, P., et al.: Bottom-up and top-down attention for image captioning and visual question answering. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)Google Scholar
  2. 2.
    Aneja, J., Deshpande, A., Schwing, A.G.: Convolutional image captioning. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)Google Scholar
  3. 3.
    Bengio, S., Vinyals, O., Jaitly, N., Shazeer, N.: Scheduled sampling for sequence prediction with recurrent neural networks. In: Advances in Neural Information Processing Systems (NeurIPS) (2015)Google Scholar
  4. 4.
    Dauphin, Y., Fan, A., Auli, M., Grangier, D.: Language modeling with gated convolutional networks. In: Proceedings of the International Conference on Machine Learning (ICML) (2017)Google Scholar
  5. 5.
    Denkowski, M.J., Lavie, A.: Meteor universal: language specific translation evaluation for any target language. In: ACL (2014)Google Scholar
  6. 6.
    Gehring, J., Auli, M., Grangier, D., Yarats, D., Dauphin, Y.: Convolutional sequence to sequence learning. In: Proceedings of the International Conference on Machine Learning (ICML) (2017)Google Scholar
  7. 7.
    Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: AISTATS 2010 (2010)Google Scholar
  8. 8.
    Gu, J., Cai, J., Wang, G., Chen, T.: Stack-captioning: coarse-to-fine learning for image captioning. In: Proceedings of the Conference on Artificial Intelligence (AAAI) (2018)Google Scholar
  9. 9.
    Gu, J., Wang, G., Cai, J., Chen, T.: An empirical study of language CNN for image captioning. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1231–1240 (2017)Google Scholar
  10. 10.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)Google Scholar
  11. 11.
    Hossain, M.Z., Sohel, F.A., Shiratuddin, M.F., Laga, H.: A comprehensive survey of deep learning for image captioning. CoRR abs/1810.04020 (2018)Google Scholar
  12. 12.
    Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3128–3137 (2015)Google Scholar
  13. 13.
    Lin, C.Y.: Rouge: a package for automatic evaluation of summaries. In: ACL (2004)Google Scholar
  14. 14.
    Lin, X., et al.: Actor-critic sequence training for image captioning. CoRR abs/1706.09601 (2017)Google Scholar
  15. 15.
    Lu, J., Xiong, C., Parikh, D., Socher, R.: Knowing when to look: adaptive attention via a visual sentinel for image captioning. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3242–3250 (2017)Google Scholar
  16. 16.
    van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., Kavukcuoglu, K.: Conditional image generation with PixelCNN decoders. In: Advances in Neural Information Processing Systems (NeurIPS) (2016)Google Scholar
  17. 17.
    Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: ACL (2002)Google Scholar
  18. 18.
    Rennie, S.J., Marcheret, E., Mroueh, Y., Ross, J., Goel, V.: Self-critical sequence training for image captioning. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1179–1195 (2017)Google Scholar
  19. 19.
    Shen, Y., Tan, S., Sordoni, A., Courville, A.: Ordered neurons: integrating tree structures into recurrent neural networks. In: International Conference on Learning Representations (2019). https://openreview.net/forum?id=B1l6qiR5F7
  20. 20.
    Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems (NeurIPS) (2017)Google Scholar
  21. 21.
    Vedantam, R., Zitnick, C.L., Parikh, D.: Cider: consensus-based image description evaluation. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4566–4575 (2015)Google Scholar
  22. 22.
    Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: a neural image caption generator. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3156–3164 (2015)Google Scholar
  23. 23.
    Wang, L., Yao, J., Tao, Y., Zhong, L., Liu, W., Du, Q.: A reinforced topic-aware convolutional sequence-to-sequence model for abstractive text summarization. In: Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI) (2018)Google Scholar
  24. 24.
    Wang, Q., Chan, A.B.: CNN+CNN: convolutional decoders for image captioning. CoRR abs/1805.09019 (2018)Google Scholar
  25. 25.
    Wang, Q., Chan, A.B.: Gated hierarchical attention for image captioning. CoRR abs/1810.12535 (2018)Google Scholar
  26. 26.
    Xu, K., et al.: Show, attend and tell: neural image caption generation with visual attention. In: Proceedings of the International Conference on Machine Learning (ICML) (2015)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  1. 1.College of Computer Science and Technology, MIIT Key Laboratory of Pattern Analysis and Machine IntelligenceNanjing University of Aeronautics and AstronauticsNanjingChina

Personalised recommendations