Abstract
Multi-label image classification focuses on predicting a set of object labels presented in an image, which is a practical yet challenging task since labels are normally co-occurred in an image while mutual interactions among labels and the correspondence between a given image and the corresponding labels are rarely considered in the existing methods. To address above challenges, we propose a Multiple Semantic Embedding model with Graph Convolutional Networks (MSEGCN) for multi-label image classification by capturing important dependencies related labels. Specifically, the proposed MSEGCN leverages graph structure to guide label co-occurrence propagation among different categories to obtain appropriate label representations. Then, by formulating multi-label classification problem as a label ranking problem with the aid of end-to-end convolutional neural network framework, we focus on learning a transform matrix to seek the image-label relevance relations in embedding space. Furthermore, an adaptive weighting strategy is introduced to effectively improve the classification performance. Experimental studies across a wide range of benchmark datasets show that our method achieves highly competitive performances against other state-of-the-art approaches.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chen, T., Wang, Z., Li, G., Lin, L.: Recurrent attentional reinforcement learning for multi-label image recognition. In: AAAI, pp. 6730–6737 (2018)
Chen, T., Xu, M., Hui, X., Wu, H., Lin, L.: Learning semantic-specific graph representation for multi-label image recognition. In: ICCV, pp. 522–531 (2019)
Chen, Z., Wei, X., Wang, P., Guo, Y.: Multi-label image recognition with graph convolutional networks. In: CVPR, pp. 5177–5186 (2019)
Chua, T., Pung, H., Lu, G., Jong, H.: A concept-based image retrieval system. In: HICSS, pp. 590–598 (1994)
Chua, T., Tang, J., Hong, R., Li, H., Luo, Z., Zheng, Y.: Nus-wide: a real-world web image database from national university of Singapore. In: CIVR, pp. 1–9 (2009)
Deng, J., Dong, W., Socher, R., Li, L., Li, K., Li, F.: Imagenet: a large-scale hierarchical image database. In: CVPR, pp. 248–255 (2009)
Frome, A., et al.: Devise: a deep visual-semantic embedding model. In: NIPS, pp. 2121–2129 (2013)
Gong, Y., Jia, Y., Leung, T., Toshev, A., Ioffe, S.: Deep convolutional ranking for multilabel image annotation. In: ICLR (2013)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)
Huynh, D., Elhamifar, E.: A shared multi-attention framework for multi-label zero-shot learning. In: CVPR, pp. 8773–8783 (2020)
Kipf, T., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)
Lee, C., Fang, W., Yeh, C., Wang, Y.: Multi-label zero-shot learning with structured knowledge graphs. In: CVPR, pp. 1576–1585 (2018)
Li, Y., Song, Y., Luo, J.: Improving pairwise ranking for multi-label image classification. In: CVPR, pp. 1837–1845 (2017)
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J.: Distributed representations of words and phrases and their compositionality. In: NIPS 26, pp. 3111–3119 (2013)
Pennington, J., Socher, R., Manning, C.: Glove: global vectors for word representation. In: EMNLP, pp. 1532–1543 (2014)
Ren, Z., Jin, H., Lin, Z., Fang, C., Yuille, A.: Multiple instance visual-semantic embedding. In: British Machine Vision Conference (2017)
Shao, J., Loy, C., Kang, K., Wang, X.: Slicing convolutional neural network for crowd video understanding. In: CVPR, pp. 5620–5628 (2016)
Wang, J., Yang, Y., Mao, J., Huang, Z., Huang, C., Xu, W.: CNN-RNN: a unified framework for multi-label image classification. In: CVPR, pp. 2285–2294 (2016)
Wang, M., Luo, C., Hong, R., Tang, J., Feng, J.: Beyond object proposals: random crop pooling for multi-label image recognition. IEEE Trans. Image Process. 25(12), 5678–5688 (2016)
Wang, Y., et al.: Multi-label classification with label graph superimposing. In: AAAI, pp. 12265–12272 (2020)
Wang, Z., Chen, T., Li, G., Xu, R., Lin, L.: Multi-label image recognition by recurrently discovering attentional regions. In: ICCV, pp. 464–472 (2017)
Yazici, V., Garcia, A., Ramisa, A., Twardowski, B., Weijer, J.: Orderless recurrent models for multi-label classification. In: CVPR, pp. 13437–13446 (2020)
Yeh, M., Li, Y.: Multilabel deep visual-semantic embedding. IEEE Trans. Pattern Anal. Mach. Intell. 42(6), 1530–1536 (2019)
Zhang, Y., Gong, B., Shah, M.: Fast zero-shot image tagging. In: CVPR, pp. 5985–5994 (2016)
Acknowledgement
This work was supported in part by the National Natural Science Foundation of China (No. 61872032) and the Beijing Natural Science Foundation (No. 4202058).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhou, T., Feng, S. (2021). Multiple Semantic Embedding with Graph Convolutional Networks for Multi-Label Image Classification. In: Ma, H., et al. Pattern Recognition and Computer Vision. PRCV 2021. Lecture Notes in Computer Science(), vol 13020. Springer, Cham. https://doi.org/10.1007/978-3-030-88007-1_37
Download citation
DOI: https://doi.org/10.1007/978-3-030-88007-1_37
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-88006-4
Online ISBN: 978-3-030-88007-1
eBook Packages: Computer ScienceComputer Science (R0)