Skip to main content
SpringerLink
Log in

Robust fine-grained image classification with noisy labels

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Since annotating fine-grained labels requires special expertise, label annotations often lack quality for many real-world fine-grained image classifications (FGIC). Due to the effectiveness of noisy labels, training deep fine-grained models directly tends to have an inferior recognition ability. To address this problem in FGIC, a robust classification approach combining “active–passive–loss (APL)” framework and multi-branch attention learning is proposed. First, in order to learn discriminative regions for classification effectively, the multi-branch attention learning framework that consists of raw, object, and part branch is introduced. These three branches are connected by attention mechanism, which enables the network to learn fine-grained features of different parts from different scales including raw, object and part levels. Second, treating noisy labels as anomalies, the novel loss framework APL that can guarantee robustness and sufficient learning is adopted to achieve robust predictions in each branch. Third, in determining the final predictions, the outputs from global and object branches are combined to achieve higher classification performance. Several experiments on fine-grained image datasets show that the proposed approach is noise-robust and can achieve excellent classification performance in the presence of noisy labels in FGIC.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Chang, D., et al.: The devil is in the channels: mutual-channel loss for fine-grained image classification. IEEE Trans. Image Process. 29, 4683–4695 (2020)

    Article  MATH  Google Scholar 

  2. Wei, X.-S., Wu, J., Cui, Q.: Deep learning for fine-grained image analysis: a survey. arXiv:1907.03069 (2019)

  3. Xie, S., Yang, T., Wang, X., Lin, Y.: Hyper-class augmented and regularized deep learning for fine-grained image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2645–2654 (2015)

  4. Peng, Y., He, X., Zhao, J.: Object-part attention model for fine-grained image classification. IEEE Trans. Image Process. 27(3), 1487–1500 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  5. Fu, J., Zheng, H., Mei, T.: Look closer to see better: RECURRENT attention convolutional neural network for fine-grained image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4438–4446 (2017)

  6. Chen, Y., Bai, Y., Zhang, W., Mei, T.: Destruction and construction learning for fine-grained image recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5157–5166 (2019)

  7. Zhang, F., Li, M., Zhai, G., Liu, Y.: Multi-branch and multi-scale attention learning for fine-grained visual categorization. In: International Conference on Multimedia Modeling, pp. 136–147. Springer (2021)

  8. Hanselmann, H., Ney, H.: ELoPE: fine-grained visual classification with efficient localization, pooling and embedding. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1247–1256 (2020)

  9. Eshratifar, A.E., Eigen, D., Gormish, M., Pedram, M.: Coarse2Fine: a two-stage training method for fine-grained visual classification. Mach. Vis. Appl. 32(2), 1–9 (2021)

    Article  Google Scholar 

  10. Zhang, N., Donahue, J., Girshick, R., Darrell, T.: Part-based R-CNNs for fine-grained category detection. In: European Conference on Computer Vision, pp. 834–849. Springer (2014)

  11. Yang, Z., Luo, T., Wang, D., Hu, Z., Gao, J., Wang, L.: Learning to navigate for fine-grained classification. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 420–435 (2018)

  12. Rodríguez, P., Velazquez, D., Cucurull, G., Gonfaus, J.M., Roca, F.X., Gonzàlez, J.: Pay attention to the activations: a modular attention mechanism for fine-grained image recognition. IEEE Trans. Multimed. 22(2), 502–514 (2019)

    Article  Google Scholar 

  13. Algan, G., Ulusoy, I.: Image classification with deep learning in the presence of noisy labels: a survey. Knowl.-Based Syst. 215, 106771 (2021)

    Article  Google Scholar 

  14. Karimi, D., Dou, H., Warfield, S.K., Gholipour, A.: Deep learning with noisy labels: exploring techniques and remedies in medical image analysis. Med. Image Anal. 65, 101759 (2020)

    Article  Google Scholar 

  15. Sun, X., Chen, L., Yang, J.: Learning from web data using adversarial discriminative neural networks for fine-grained classification. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, pp. 273–280 (2019)

  16. Ghosh, A., Kumar, H., Sastry, P.: Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31, no. 1 (2017)

  17. Yi, K., Wu, J.: Probabilistic end-to-end noise correction for learning with noisy labels. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7017–7025 (2019)

  18. Tanaka, D., Ikami, D., Yamasaki, T., Aizawa, K.: Joint optimization framework for learning with noisy labels. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5552–5560 (2018)

  19. Wang, Y., Ma, X., Chen, Z., Luo, Y., Yi, J., Bailey, J.: Symmetric cross entropy for robust learning with noisy labels. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 322–330 (2019)

  20. Li, J., Wong, Y., Zhao, Q., Kankanhalli, M.S.: Learning to learn from noisy labeled data. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5051–5059 (2019)

  21. Zhang, Z., Sabuncu, M.R.: Generalized cross entropy loss for training deep neural networks with noisy labels. In: 32nd Conference on Neural Information Processing Systems (NeurIPS) (2018)

  22. Xu, Y., Cao, P., Kong, Y., Wang, Y.: L_DMI: a novel information-theoretic loss function for training deep nets robust to label noise. In: NeurIPS, pp. 6222–6233 (2019)

  23. Ma, X., Huang, H., Wang, Y., Romano, S., Erfani, S., Bailey, J.: Normalized loss functions for deep learning with noisy labels. In: International Conference on Machine Learning, pp. 6543–6553. PMLR (2020)

  24. Wei, X.-S., Xie, C.-W., Wu, J., Shen, C.: Mask-CNN: localizing parts and selecting descriptors for fine-grained bird species categorization. Pattern Recogn. 76, 704–714 (2018)

    Article  Google Scholar 

  25. He, X., Peng, Y.: Weakly supervised learning of part selection model with spatial constraints for fine-grained image classification. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31, no. 1 (2017)

  26. Lin, T.-Y., RoyChowdhury, A., Maji, S.: Bilinear convolutional neural networks for fine-grained visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 40(6), 1309–1322 (2017)

    Article  Google Scholar 

  27. Dubey, A., Gupta, O., Raskar, R., Naik, N.: Maximum-entropy fine grained classification. In: Advances in Neural Information Processing Systems, vol. 31 (2018)

  28. Han, J., Yao, X., Cheng, G., Feng, X., Xu, D.; P-CNN: part-based convolutional neural networks for fine-grained visual categorization. IEEE Trans. Pattern Anal. Mach. Intell 44(2), 579–590 (2022). https://doi.org/10.1109/TPAMI.2019.2933510

    Article  Google Scholar 

  29. Li, X., Wu, J., Sun, Z., Ma, Z., Cao, J., Xue, J.-H.: BSNet: bi-similarity network for few-shot fine-grained image classification. IEEE Trans. Image Process. 30, 1318–1331 (2020)

    Article  MathSciNet  Google Scholar 

  30. Niu, L., Veeraraghavan, A., Sabharwal, A.: Webly supervised learning meets zero-shot learning: a hybrid approach for fine-grained classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7171–7180 (2018)

  31. Cheng, G., Li, R., Lang, C., Han, J.: Task-wise attention guided part complementary learning for few-shot image classification. Sci. China Inf. Sci. 64(2), 1–14 (2021)

    Article  Google Scholar 

  32. He, X., Peng, Y.: Fine-grained image classification via combining vision and language. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5994–6002 (2017)

  33. Chen, P., Liao, B.B., Chen, G., Zhang, S.: Understanding and utilizing deep neural networks trained with noisy labels. In: International Conference on Machine Learning, pp. 1062–1070. PMLR (2019)

  34. Jiang, L., Huang, D., Liu, M., Yang, W.: Beyond synthetic noise: deep learning on controlled noisy labels. In: International Conference on Machine Learning, pp. 4804–4815. PMLR (2020)

  35. Li, J., Socher, R., Hoi, S.C.: DivideMix: learning with noisy labels as semi-supervised learning. In: International Conference on Learning Representations (2019)

  36. Wang, Z., Hu, G., Hu, Q.: Training noise-robust deep neural networks via meta-learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4524–4533 (2020)

  37. Jindal, I., Nokleby, M., Chen, X.: Learning deep networks from noisy labels with dropout regularization. In: 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 967–972. IEEE (2016)

  38. Ahmed, A., Yousif, H., He, Z.: Ensemble diversified learning for image classification with noisy labels. Multimed. Tools Appl. 80, 1–14 (2021)

    Article  Google Scholar 

  39. Pang, T., Du, C., Zhu, J.: Robust deep learning via reverse cross-entropy training and thresholding test, vol. 3. arXiv:1706.00633 (2017)

  40. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. IEEE (2016)

  41. Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The caltech-ucsd birds-200-2011 dataset (2011)

  42. Krause, J., Stark, M., Deng, J., Fei-Fei, L.: 3d object representations for fine-grained categorization. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 554–561 (2013)

  43. Song, H., Kim, M., Lee, J.-G.: Selfie: refurbishing unclean samples for robust deep learning. In: International Conference on Machine Learning, pp. 5907–5915. PMLR (2019)

  44. Lee, K.-H., He, X., Zhang, L., Yang, L.: Cleannet: transfer learning for scalable image classifier training with label noise. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5447–5456 (2018)

  45. Chen, Y., Shen, X., Hu, S.X., Suykens, J.A.: Boosting co-teaching with compression regularization for label noise. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2688–2692 (2021)

  46. Feng, C., Tzimiropoulos, G., Patras, I.: S3: supervised self-supervised learning under label noise. arXiv:2111.11288 (2021)

  47. Li, Q., et al.: Product image recognition with guidance learning and noisy supervision. Comput. Vis. Image Underst. 196, 102963 (2020)

    Article  Google Scholar 

  48. Cordeiro, F.R., Sachdeva, R., Belagiannis, V., Reid, I., Carneiro, G.: Longremix: robust learning with high confidence samples in a noisy label environment. arXiv:2103.04173 (2021)

Download references

Acknowledgements

The work is supported by the Fundamental Research Funds for the Central Universities (Grant No. 2020-IB-003)

Author information

Authors and Affiliations

  1. Department of Information Engineering, Wuhan Institute of City, Wuhan, 430083, China

    Xinxing Tan

  2. Research and Training Center, Wuhan Institute of City, Wuhan, 430083, China

    Zemin Dong

  3. School of Science, Wuhan University of Technology, Wuhan, 430070, China

    Hualing Zhao

Authors
  1. Xinxing Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zemin Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hualing Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zemin Dong.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tan, X., Dong, Z. & Zhao, H. Robust fine-grained image classification with noisy labels. Vis Comput 39, 5637–5650 (2023). https://doi.org/10.1007/s00371-022-02686-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02686-w

Keywords

Search

Navigation