Abstract
A convolutional decoder for image caption has proven to be easier to train than the Long Short Term Memory (LSTM) decoder [2]. However, previous convolutional image captioning methods are not good at capture the relationship between generated words, which could lead to failure in predicting visually unrelated words. To address this issue, we propose an Adaptive Joint Attention (AJA) module for the convolutional decoder, by incorporating self-attention to explore the correlation between generated words automatically. Specifically, we develop a word gate with multi-head self-attention to directly capture words’ relationship. An adaptive combination strategy is proposed to learn which word is less related to image information. Besides, reinforcement learning is applied for directly optimizing the non-differentiable metrics while avoiding the exposure bias during inference. Extensive evaluations on two public data sets demonstrate that our method outperforms several state-of-the-art approaches in image captioning task.
This research was supported by the National Natural Science Foundation of China (61876082, 61861130366, 61732006, 61473149).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Anderson, P., et al.: Bottom-up and top-down attention for image captioning and visual question answering. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
Aneja, J., Deshpande, A., Schwing, A.G.: Convolutional image captioning. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
Bengio, S., Vinyals, O., Jaitly, N., Shazeer, N.: Scheduled sampling for sequence prediction with recurrent neural networks. In: Advances in Neural Information Processing Systems (NeurIPS) (2015)
Dauphin, Y., Fan, A., Auli, M., Grangier, D.: Language modeling with gated convolutional networks. In: Proceedings of the International Conference on Machine Learning (ICML) (2017)
Denkowski, M.J., Lavie, A.: Meteor universal: language specific translation evaluation for any target language. In: ACL (2014)
Gehring, J., Auli, M., Grangier, D., Yarats, D., Dauphin, Y.: Convolutional sequence to sequence learning. In: Proceedings of the International Conference on Machine Learning (ICML) (2017)
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: AISTATS 2010 (2010)
Gu, J., Cai, J., Wang, G., Chen, T.: Stack-captioning: coarse-to-fine learning for image captioning. In: Proceedings of the Conference on Artificial Intelligence (AAAI) (2018)
Gu, J., Wang, G., Cai, J., Chen, T.: An empirical study of language CNN for image captioning. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1231–1240 (2017)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)
Hossain, M.Z., Sohel, F.A., Shiratuddin, M.F., Laga, H.: A comprehensive survey of deep learning for image captioning. CoRR abs/1810.04020 (2018)
Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3128–3137 (2015)
Lin, C.Y.: Rouge: a package for automatic evaluation of summaries. In: ACL (2004)
Lin, X., et al.: Actor-critic sequence training for image captioning. CoRR abs/1706.09601 (2017)
Lu, J., Xiong, C., Parikh, D., Socher, R.: Knowing when to look: adaptive attention via a visual sentinel for image captioning. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3242–3250 (2017)
van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., Kavukcuoglu, K.: Conditional image generation with PixelCNN decoders. In: Advances in Neural Information Processing Systems (NeurIPS) (2016)
Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: ACL (2002)
Rennie, S.J., Marcheret, E., Mroueh, Y., Ross, J., Goel, V.: Self-critical sequence training for image captioning. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1179–1195 (2017)
Shen, Y., Tan, S., Sordoni, A., Courville, A.: Ordered neurons: integrating tree structures into recurrent neural networks. In: International Conference on Learning Representations (2019). https://openreview.net/forum?id=B1l6qiR5F7
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems (NeurIPS) (2017)
Vedantam, R., Zitnick, C.L., Parikh, D.: Cider: consensus-based image description evaluation. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4566–4575 (2015)
Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: a neural image caption generator. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3156–3164 (2015)
Wang, L., Yao, J., Tao, Y., Zhong, L., Liu, W., Du, Q.: A reinforced topic-aware convolutional sequence-to-sequence model for abstractive text summarization. In: Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI) (2018)
Wang, Q., Chan, A.B.: CNN+CNN: convolutional decoders for image captioning. CoRR abs/1805.09019 (2018)
Wang, Q., Chan, A.B.: Gated hierarchical attention for image captioning. CoRR abs/1810.12535 (2018)
Xu, K., et al.: Show, attend and tell: neural image caption generation with visual attention. In: Proceedings of the International Conference on Machine Learning (ICML) (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Chen, R., Li, Z., Zhang, D. (2019). Adaptive Joint Attention with Reinforcement Training for Convolutional Image Caption. In: Zeng, A., Pan, D., Hao, T., Zhang, D., Shi, Y., Song, X. (eds) Human Brain and Artificial Intelligence. HBAI 2019. Communications in Computer and Information Science, vol 1072. Springer, Singapore. https://doi.org/10.1007/978-981-15-1398-5_17
Download citation
DOI: https://doi.org/10.1007/978-981-15-1398-5_17
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-1397-8
Online ISBN: 978-981-15-1398-5
eBook Packages: Computer ScienceComputer Science (R0)