Advertisement

Zero-shot Fine-grained Classification by Deep Feature Learning with Semantics

  • Ao-Xue LiEmail author
  • Ke-Xin Zhang
  • Li-Wei Wang
Research Article

Abstract

Fine-grained image classification, which aims to distinguish images with subtle distinctions, is a challenging task for two main reasons: lack of sufficient training data for every class and difficulty in learning discriminative features for representation. In this paper, to address the two issues, we propose a two-phase framework for recognizing images from unseen fine-grained classes, i.e., zero-shot fine-grained classification. In the first feature learning phase, we finetune deep convolutional neural networks using hierarchical semantic structure among fine-grained classes to extract discriminative deep visual features. Meanwhile, a domain adaptation structure is induced into deep convolutional neural networks to avoid domain shift from training data to test data. In the second label inference phase, a semantic directed graph is constructed over attributes of fine-grained classes. Based on this graph, we develop a label propagation algorithm to infer the labels of images in the unseen classes. Experimental results on two benchmark datasets demonstrate that our model outperforms the state-of-the-art zero-shot learning models. In addition, the features obtained by our feature learning model also yield significant gains when they are used by other zero-shot learning models, which shows the flexility of our model in zero-shot fine-grained classification.

Keywords

Fine-grained image classification zero-shot learning deep feature learning domain adaptation semantic graph 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Notes

Acknowledgement

This work was supported by National Basic Research Program of China (973 Program) (No. 2015CB352502), National Nature Science Foundation of China (No. 61573026) and Beijing Nature Science Foundation (No. L172037).

References

  1. [1]
    B. Zhao, J. S. Feng, X. Wu, S. C. Yan. A survey on deep learning-based fine-grained object classification and semantic segmentation. International Journal of Automation and Computing, vol. 14, no. 2, pp. 119–135, 2017. DOI:  https://doi.org/10.1007/s11633-017-1053-3.CrossRefGoogle Scholar
  2. [2]
    M. El Mallahi, A. Zouhri, A. El Affar, A. Tahiri, H. Qjidaa. Radial Hahn moment invariants for 2D and 3D image recognition. International Journal of Automation and Computing, vol. 15, no. 3, pp. 277–289, 2018. DOI:  https://doi.org/10.1007/s11633-017-1071-1.CrossRefGoogle Scholar
  3. [3]
    H. S. Du, Q. P. Hu, D. F. Qiao, I Pitas. Robust face recognition via low-rank sparse representation-based classification. International Journal of Automation and Computing, vol. 12, no. 6, pp. 579–587, 2015. DOI:  https://doi.org/10.1007/s11633-015-0901-2.CrossRefGoogle Scholar
  4. [4]
    T. Long, X. Xu, F. M. Shen, L. Liu, N. Xie, Y. Yang. Zero-shot learning via discriminative representation extraction. Pattern Recognition Letters, vol. 109, pp. 27–34, 2018. DOI:  https://doi.org/10.1016/j.patrec.2017.09.030.CrossRefGoogle Scholar
  5. [5]
    E. Kodirov, T. Xiang, S. G. Gong. Semantic autoencoder for zero-shot learning. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, pp. 3174–3183, 2017. DOI: https://doi.org/18.1109/CVPR.2017.473.Google Scholar
  6. [6]
    C. Wah, S. Branson, P. Welinder, P. Perona, S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset, Technical Report CNS-TR-2011-001, California Institute of Technology, USA, 2011.Google Scholar
  7. [7]
    P. Welinder, S. Branson, T. Mita, C Wah, F. Schroff, S. Belongie, P. Perona. Caltech-UCSD Birds 200, Technical Report CNS-TR-2010-001, California Institute of Technology, USA, 2010.Google Scholar
  8. [8]
    T. Berg, J. X. Liu, S. W. Lee, M. L. Alexander, D. W. Jacobs, P. N. Belhumeur. Birdsnap: Large-scale fine-grained visual categorization of birds. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Columbus, USA, pp. 2019–2026, 2014. doi  https://doi.org/10.1109/CVPR.2014.259.
  9. [9]
    B. P. Yao, A. Khosla, F. F. Li. Combining randomization and discrimination for fine-grained image categorization. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Springs, Colorado, USA, pp. 1577–1584, 2011.Google Scholar
  10. [10]
    M. E. Nilsback, A. Zisserman. Automated flower classification over a large number of classes. In Proceedings of the 6th Indian Conference on Computer Vision, Graphics & Image Processing, IEEE, Bhubaneswar, India, pp. 722–729, 2008. DOI:  https://doi.org/10.1109/ICVGIP.2008.47.Google Scholar
  11. [11]
    A. R. Sfar, N. Boujemaa, D. Geman. Vantage feature frames for fine-grained categorization. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Portland, USA, pp. 835–842, 2013. DOI:  https://doi.org/10.1109/CVPR2013113.
  12. [12]
    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. H. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, L. Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, vol. 115, no. 3, pp. 211–252, 2015. DOI:  https://doi.org/10.1007/s11263-015-0816-y.MathSciNetCrossRefGoogle Scholar
  13. [13]
    B. Romera-Paredes, P. H. S. Torr. An embarrassingly simple approach to zero-shot learning. In Proceedings of the 32nd International Conference on Machine Learning, ACM, Lille, France, pp. 2152–2161, 2015.Google Scholar
  14. [14]
    Z. M. Zhang, V. Saligrama. Zero-shot learning via semantic similarity embedding. In Proceedings of IEEE International Conference on Computer Vision, Santiago, Chile, pp. 4166–1174, 2015. DOI:  https://doi.org/10.1109/ICCV.2015.474.
  15. [15]
    Z. Y. Fu, T. A. Xiang, E. Kodirov, S. G. Gong. Zero-shot object recognition by semantic manifold distance. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, pp. 2635–2644, 2015. DOI:  https://doi.org/10.1109/CVPR.2015.7298879.
  16. [16]
    T. Mikolov, I. Sutskever, K. Chen, G. Corrado, J. Dean. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems, Curran Associates Inc., Lake Tahoe, USA, spp. 1188–1196, 2013.Google Scholar
  17. [17]
    N. Zhang, J. Donahue, R. Girshick, T. Darrell. Part-based R-CNNs for fine-grained category detection. In Proceedings of the 13th European Conference on Computer Vision, Springer, Zurich, Switzerland, pp. 834–849, 2014. DOI:  https://doi.org/10.1007/978-3-319-10590-1.54.Google Scholar
  18. [18]
    S. L. Huang, Z. Xu, D. C. Tao, Y. Zhang. Part-stacked CNN for fine-grained visual categorization. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 1173–1182, 2016. DOI:  https://doi.org/10.1109/CVPR.2016.132.
  19. [19]
    A. Krizhevsky, I. Sutskever, G. E. Hinton. ImageNet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems, Curran Associates Inc., Lake Tahoe, USA, pp. 1097–1105, 2012.Google Scholar
  20. [20]
    Z. Xu, S. L. Huang, Y. Zhang, D. C. Tao. Augmenting strong supervision using web data for fine-grained categorization. In Proceedings of IEEE International Conference on Computer Vision, Santiago, Chile, pp. 2524–2532, 2015. DOI:  https://doi.org/10.1109/ICCV.2015.290.
  21. [21]
    H. Zhang, T. Xu, M. Elhoseiny, X. L. Huang, S. T. Zhang, A. Elgammal, D. Metaxas. SPDA-CNN: Unifying semantic part detection and abstraction for fine-grained recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 1143–1152, 2016. DOI:  https://doi.org/10.1109/CVPR.2016.129.
  22. [22]
    C. Szegedy, W. Liu, Y. Q. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich. Going deeper with convolutions. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, pp. 1–9, 2015. DOI:  https://doi.org/10.1109/CVPR.2015.7298594.
  23. [23]
    T. Y. Lin, A. RoyChowdhury, S. Maji. Bilinear CNN models for fine-grained visual recognition. In Proceedings of IEEE International Conference on Computer Vision, Santiago, Chile, pp. 1449–1457, 2015. DOI:  https://doi.org/10.1109/ICCV.2015.170.
  24. [24]
    X. P. Zhang, H. K. Xiong, W. G. Zhou, W. Y. Lin, Q. Tian. Picking deep filter responses for fine-grained image recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 1134–1142, 2016. DOI:  https://doi.org/10.1109/CVPR.2016.128.
  25. [25]
    E. Kodirov, T. Xiang, Z. Y. Fu, S. G. Gong. Unsupervised domain adaptation for zero-shot learning. In Proceedings of IEEE International Conference on Computer Vision, Santiago, Chile, pp. 2452–2460, 2015. DOI:  https://doi.org/10.1109/ICCV.2015.282.
  26. [26]
    C. H. Lampert, H. Nickisch, S. Harmeling. Attribute-based classification for zero-shot visual object categorization. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 36, no. 3, pp. 453–465, 2014. DOI:  https://doi.org/10.1109/TPAMI.2013.140.CrossRefGoogle Scholar
  27. [27]
    P. Kankuekul, A. Kawewong, S. Tangruamsub, O. Hasegawa. Online incremental attribute-based zero-shot learning. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Providence, USA, pp. 3657–3664, 2012. DOI:  https://doi.org/10.1109/CVPR.2012.6248112.
  28. [28]
    M. Rohrbach, M. Stark, B. Schiele. Evaluating knowledge transfer and zero-shot learning in a large-scale setting. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Springs, Colorado USA, pp. 1641–1648, 2011. DOI:  https://doi.org/10.1109/CVPR.2011.5995627.
  29. [29]
    X. D. Yu, Y. Aloimonos. Attribute-based transfer learning for object categorization with zero/one training example. In Proceedings of the 11th European Conference on Computer Vision, Springer, Heraklion, Greece, pp. 127–140, 2010. DOI:  https://doi.org/10.1007/978-3-642-15555-0_10.Google Scholar
  30. [30]
    M. Palatucci, D. Pomerleau, G. Hinton, T. M. Mitchell. Zero-shot learning with semantic output codes. In Proceedings of the 22nd International Conference on Neural Information Processing Systems, Curran Associates Inc., Vancouver, Canada, pp. 1410–1418, 2009.Google Scholar
  31. [31]
    C. H. Lampert, H. Nickisch, S. Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Miami, USA, pp. 951–958, 2009. DOI:  https://doi.org/10.1109/CVPR.2009.5206594.
  32. [32]
    Y. Q. Xian, B. Schiele, Z. Akata. Zero-shot learning-the good, the bad and the ugly. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, pp. 4582–4591, 2017. DOI:  https://doi.org/10.1109/CVPR.2017.328.
  33. [33]
    Z. M. Zhang, V. Saligrama. Zero-shot learning via joint latent similarity embedding. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 6034–6042, 2016. DOI:  https://doi.org/10.1109/CVPR.2016.649.
  34. [34]
    M. Bucher, S. Herbin, F. Jurie. Improving semantic embedding consistency by metric learning for zero-shot classification. In Proceedings of the 14th European Conference on Computer Vision, Springer, Amsterdam, The Netherlands, pp. 730–746, 2016. DOI:  https://doi.org/10.1007/978-3-319-46454-1_44.Google Scholar
  35. [35]
    Z. Akata, M. Malinowski, M. Fritz, B. Schiele. Multi-cue zero-shot learning with strong supervision. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Las Vegas, USA, pp. 59–68, 2016. DOI:  https://doi.org/10.1109/CVPR.2016.14.Google Scholar
  36. [36]
    R. Z. Qiao, L. Q. Liu, C. H. Shen, A. van denHengel. Less is more: Zero-shot learning from online textual documents with noise suppression. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 2441–2448, 2016. DOI:  https://doi.org/10.1109/CVPR.2016.247.
  37. [37]
    K. Simonyan, A. Zisserman. Very deep convolutional networks for large-scale image recognition. In Proceedings of International Conference on Learning Representations, ICLR, San Diego, USA, pp. 59–68, 2015.Google Scholar
  38. [38]
    Y. Ganin, V. S. Lempitsky. Unsupervised domain adaptation by backpropagation. In Proceedings of the 32nd International Conference on Machine Learning, ICML, Lille, France, pp. 1180–1189, 2015.Google Scholar
  39. [39]
    D. Y. Zhou, J. Y. Huang, B. Scholkopf. Learning from labelled and unlabelled data on a directed graph. In Proceedings of the 22nd International Conference on Machine Learning, ICML, Bonn, Germany, pp. 1036–1043, 2005.Google Scholar
  40. [40]
    A. X. Li, Z. W. Lu, L. W. Wang, T. Xiang, J. R. Wen. Zero-shot scene classification for high spatial resolution remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, vol. 55, no. 7, pp. 4157–4167, 2017. DOI:  https://doi.org/10.1109/TGRS.2017.2689071.CrossRefGoogle Scholar
  41. [41]
    R. E. Fan, K. W. Chang, C. J. Hsieh, X. R. Wang, C. J. Lin. LIBLINEAR: A library for large linear classification. Journal of Machine Learning Research, vol. 9, pp. 1871–1874, 2008.zbMATHGoogle Scholar
  42. [42]
    Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, vol. 1, no. 4, pp. 541–551, 1989. DOI:  https://doi.org/10.1162/neco.1989.1.4.541.CrossRefGoogle Scholar
  43. [43]
    Y. Q. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, T. Darrell. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM International Conference on Multimedia, ACM, Orlando, USA, 2014. DOI:  https://doi.org/10.1145/2647868.2654889.Google Scholar

Copyright information

© Institute of Automatinn, Chinese Academy of Sciences and Springer-Verlag Gmbh Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.The Key Laboratory of Machine Perception (MOE), School of Electronics Engineering and Computer SciencePeking UniversityBeijingChina

Personalised recommendations