Skip to main content
SpringerLink
Log in

Multi-label Image Classification with Multi-scale Global-Local Semantic Graph Network

  • Conference paper
  • First Online:
Machine Learning and Knowledge Discovery in Databases: Research Track (ECML PKDD 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14171))

Abstract

With the development of deep learning techniques, multi-label image classification tasks have achieved good performance. Recently, graph convolutional network has been proved to be an effective way to explore the labels dependencies. However, due to the complexity of label semantic relations, the static dependencies obtained by existing methods cannot consider the overall characteristics of an image and accurately locate the target region. Therefore, we propose the Multi-scale Global-local Semantic Graph Network (MGSGN) for multi-label image classification, which mainly includes three important parts. First, the multi-scale feature reconstruction aggregates complementary information at different levels in CNN through cross-layer attention, which can effectively identify target categories of different sizes. We then design a channel dual-branch cross-attention module to explore the correlation between global information and local features in multi-scale features, which using the way of adaptive cross-fusion to locate the target area more accurately. Moreover, we propose the multi-perspective weighted cosine measure in multi-perspective dynamic semantic representation module to construct content-based label dependencies for each image to dynamically construct a semantic relationship graph. Extensive experiments on the two public datasets have verified that the classification performance of our model is better than many state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Boutell, M.R., Luo, J., Shen, X., Brown, C.M.: Learning multi-label scene classification. Pattern Recogn. 37(9), 1757–1771 (2004)

    Article  Google Scholar 

  2. Chen, S.F., Chen, Y.C., Yeh, C.K., Wang, Y.C.: Order-free RNN with visual attention for multi-label classification. In: Proceedings of the 32th AAAI Conference on Artificial Intelligence, vol. 32 (2018)

    Google Scholar 

  3. Chen, S., Li, Z., Tang, Z.: Relation R-CNN: a graph based relation-aware network for object detection. IEEE Signal Process. Lett. 27, 1680–1684 (2020)

    Article  Google Scholar 

  4. Chen, T., Lin, L., Chen, R., Hui, X., Wu, H.: Knowledge-guided multi-label few-shot learning for general image recognition. IEEE Trans. Pattern Anal. Mach. Intell. 44(3), 1371–1384 (2020)

    Article  Google Scholar 

  5. Chen, T., Wang, Z., Li, G., Lin, L.: Recurrent attentional reinforcement learning for multi-label image recognition. In: Proceedings of the 32th AAAI Conference on Artificial Intelligence, vol. 32 (2018)

    Google Scholar 

  6. Chen, T., Xu, M., Hui, X., Wu, H., Lin, L.: Learning semantic-specific graph representation for multi-label image recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 522–531 (2019)

    Google Scholar 

  7. Chen, Y., Zou, C., Chen, J.: Label-aware graph representation learning for multi-label image classification. Neurocomputing 492, 50–61 (2022)

    Article  Google Scholar 

  8. Chen, Z.M., Wei, X.S., Wang, P., Guo, Y.: Multi-label image recognition with graph convolutional networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5177–5186 (2019)

    Google Scholar 

  9. Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1251–1258 (2017)

    Google Scholar 

  10. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)

    Google Scholar 

  11. Everingham, M., Eslami, S.A., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes challenge: a retrospective. Int. J. Comput. Vision 111, 98–136 (2015)

    Article  Google Scholar 

  12. Gao, B.B., Zhou, H.Y.: Learning to discover multi-class attentional regions for multi-label image recognition. IEEE Trans. Image Process. 30, 5920–5932 (2021)

    Article  Google Scholar 

  13. Gao, P., et al.: Dynamic fusion with intra-and inter-modality attention flow for visual question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6639–6648 (2019)

    Google Scholar 

  14. Ge, Z., Mahapatra, D., Sedai, S., Garnavi, R., Chakravorty, R.: Chest X-rays classification: a multi-label and fine-grained problem. arXiv preprint arXiv:1807.07247 (2018)

  15. Guo, H., Zheng, K., Fan, X., Yu, H., Wang, S.: Visual attention consistency under image transforms for multi-label image classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 729–739 (2019)

    Google Scholar 

  16. Hassanin, M., Radwan, I., Khan, S., Tahtali, M.: Learning discriminative representations for multi-label image recognition. J. Vis. Commun. Image Represent. 83, 103448 (2022)

    Article  Google Scholar 

  17. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  18. Hu, H., Zhou, G.T., Deng, Z., Liao, Z., Mori, G.: Learning structured inference neural networks with label relations. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2960–2968 (2016)

    Google Scholar 

  19. Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)

  20. Lanchantin, J., Wang, T., Ordonez, V., Qi, Y.: General multi-label image classification with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16478–16488 (2021)

    Google Scholar 

  21. Li, Q., Peng, X., Qiao, Y., Peng, Q.: Learning category correlations for multi-label image recognition with graph networks. arXiv preprint arXiv:1909.13005 (2019)

  22. Li, X., Zhao, F., Guo, Y.: Multi-label image classification with a probabilistic label enhancement model. In: Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence, vol. 1, pp. 1–10 (2014)

    Google Scholar 

  23. Li, Y., Huang, C., Loy, C.C., Tang, X.: Human attribute recognition by deep hierarchical contexts. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 684–700. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46466-4_41

    Chapter  Google Scholar 

  24. Li, Z., Lin, L., Zhang, C., Ma, H., Zhao, W., Shi, Z.: A semi-supervised learning approach based on adaptive weighted fusion for automatic image annotation. ACM Trans. Multimedia Comput. Commun. Appl. (TOMM) 17(1), 1–23 (2021)

    Article  Google Scholar 

  25. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  26. Maas, A.L., Hannun, A.Y., Ng, A.Y., et al.: Rectifier nonlinearities improve neural network acoustic models. In: Proceedings of the 30th International Conference on Machine Learning, Atlanta, Georgia, USA, vol. 30, p. 3 (2013)

    Google Scholar 

  27. Paszke, A., et al.: Automatic differentiation in pytorch (2017)

    Google Scholar 

  28. Wang, J., Yang, Y., Mao, J., Huang, Z., Huang, C., Xu, W.: CNN-RNN: a unified framework for multi-label image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2285–2294 (2016)

    Google Scholar 

  29. Wang, Y., et al.: Multi-label classification with label graph superimposing. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence, vol. 34, pp. 12265–12272 (2020)

    Google Scholar 

  30. Wang, Z., Fang, Z., Li, D., Yang, H., Du, W.: Semantic supplementary network with prior information for multi-label image classification. IEEE Trans. Circuits Syst. Video Technol. 32(4), 1848–1859 (2021)

    Article  Google Scholar 

  31. Wang, Z., Chen, T., Li, G., Xu, R., Lin, L.: Multi-label image recognition by recurrently discovering attentional regions. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 464–472 (2017)

    Google Scholar 

  32. Xian, T., Li, Z., Tang, Z., Ma, H.: Adaptive path selection for dynamic image captioning. IEEE Trans. Circuits Syst. Video Technol. 32(9), 5762–5775 (2022)

    Article  Google Scholar 

  33. Ye, J., He, J., Peng, X., Wu, W., Qiao, Yu.: Attention-driven dynamic graph convolutional network for multi-label image recognition. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12366, pp. 649–665. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58589-1_39

    Chapter  Google Scholar 

  34. Zhao, J., Yan, K., Zhao, Y., Guo, X., Huang, F., Li, J.: Transformer-based dual relation graph for multi-label image recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 163–172 (2021)

    Google Scholar 

  35. Zhao, Q., Wang, X., Lyu, S., Liu, B., Yang, Y.: A feature consistency driven attention erasing network for fine-grained image retrieval. Pattern Recogn. 128, 108618 (2022)

    Article  Google Scholar 

  36. Zhou, F., Huang, S., Liu, B., Yang, D.: Multi-label image classification via category prototype compositional learning. IEEE Trans. Circuits Syst. Video Technol. 32(7), 4513–4525 (2021)

    Article  Google Scholar 

  37. Zhou, T., Li, Z., Zhang, C., Ma, H.: Classify multi-label images via improved CNN model with adversarial network. Multimedia Tools Appl. 79, 6871–6890 (2020)

    Article  Google Scholar 

  38. Zhou, W., Dou, P., Su, T., Hu, H., Zheng, Z.: Feature learning network with transformer for multi-label image classification. Pattern Recogn. 136, 109203 (2023)

    Article  Google Scholar 

  39. Zhou, W., Hou, Y., Chen, D., Hu, H., Su, T.: Attention-augmented memory network for image multi-label classification. ACM Trans. Multimedia Comput. Commun. Appl. 19(3), 1–24 (2022)

    Article  Google Scholar 

  40. Zhou, W., Xia, Z., Dou, P., Su, T., Hu, H.: Double attention based on graph attention network for image multi-label classification. ACM Trans. Multimed. Comput. Commun. Appl. 19(1), 1–23 (2023)

    Article  Google Scholar 

  41. Zhu, F., Li, H., Ouyang, W., Yu, N., Wang, X.: Learning spatial regularization with image-level supervisions for multi-label image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5513–5522 (2017)

    Google Scholar 

  42. Zhu, K., Wu, J.: Residual attention: a simple but effective method for multi-label recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 184–193 (2021)

    Google Scholar 

  43. Zhu, Q., Kuang, W., Li, Z.: Dual attention interactive fine-grained classification network based on data augmentation. J. Vis. Commun. Image Represent. 88, 103632 (2022)

    Article  Google Scholar 

  44. Zhu, Q., Kuang, W., Zhixin, L.: Fusing bilinear multi-channel gated vector for fine-grained classification. Mach. Vis. Appl. 34(2), 26 (2023)

    Article  Google Scholar 

Download references

Acknowledgments

This work is supported by National Natural Science Foundation of China (Nos. 62276073, 61966004), Guangxi Natural Science Foundation (No. 2019GXNSFDA245018), Innovation Project of Guangxi Graduate Education (No. YCBZ2022060), Guangxi “Bagui Scholar” Teams for Innovation and Research Project, and Guangxi Collaborative Innovation Center of Multi-source Information Integration and Intelligent Processing.

Author information

Authors and Affiliations

  1. Key Lab of Education Blockchain and Intelligent Technology, Ministry of Education, Guangxi Normal University, Guilin, 541004, China

    Wenlan Kuang, Qiangxi Zhu & Zhixin Li

  2. Guangxi Key Lab of Multi-source Information Mining and Security, Guangxi Normal University, Guilin, 541004, China

    Wenlan Kuang, Qiangxi Zhu & Zhixin Li

Authors
  1. Wenlan Kuang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qiangxi Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhixin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhixin Li .

Editor information

Editors and Affiliations

  1. University of Michigan, Ann Arbor, MI, USA

    Danai Koutra

  2. University of Vienna, Vienna, Austria

    Claudia Plant

  3. Max Planck Institute for Software Systems, Kaiserslautern, Germany

    Manuel Gomez Rodriguez

  4. Politecnico di Torino, Turin, Italy

    Elena Baralis

  5. CENTAI, Turin, Italy

    Francesco Bonchi

Ethics declarations

Ethical Statement

We affirm that the ideas, concepts, and findings presented in this paper are the result of our own original work, conducted with honesty, rigor, and transparency. We have provided proper citations and references for all sources used, and have clearly acknowledged the contributions of others where applicable.

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kuang, W., Zhu, Q., Li, Z. (2023). Multi-label Image Classification with Multi-scale Global-Local Semantic Graph Network. In: Koutra, D., Plant, C., Gomez Rodriguez, M., Baralis, E., Bonchi, F. (eds) Machine Learning and Knowledge Discovery in Databases: Research Track. ECML PKDD 2023. Lecture Notes in Computer Science(), vol 14171. Springer, Cham. https://doi.org/10.1007/978-3-031-43418-1_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-43418-1_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-43417-4

  • Online ISBN: 978-3-031-43418-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation