Advertisement

Large-Scale Object Classification Using Label Relation Graphs

  • Jia Deng
  • Nan Ding
  • Yangqing Jia
  • Andrea Frome
  • Kevin Murphy
  • Samy Bengio
  • Yuan Li
  • Hartmut Neven
  • Hartwig Adam
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8689)

Abstract

In this paper we study how to perform object classification in a principled way that exploits the rich structure of real world labels. We develop a new model that allows encoding of flexible relations between labels. We introduce Hierarchy and Exclusion (HEX) graphs, a new formalism that captures semantic relations between any two labels applied to the same object: mutual exclusion, overlap and subsumption. We then provide rigorous theoretical analysis that illustrates properties of HEX graphs such as consistency, equivalence, and computational implications of the graph structure. Next, we propose a probabilistic classification model based on HEX graphs and show that it enjoys a number of desirable properties. Finally, we evaluate our method using a large-scale benchmark. Empirical results demonstrate that our model can significantly improve object classification by exploiting the label relations.

Keywords

Object Recognition Categorization 

Supplementary material

978-3-319-10590-1_4_MOESM1_ESM.pdf (349 kb)
Electronic Supplementary Material (PDF 349 KB)

References

  1. 1.
    Akata, Z., Perronnin, F., Harchaoui, Z., Schmid, C.: Label-embedding for attribute-based classification. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 819–826. IEEE (2013)Google Scholar
  2. 2.
    Amit, Y., Fink, M., Srebro, N., Ullman, S.: Uncovering shared structures in multiclass classification. In: Proceedings of the 24th International Conference on Machine Learning, pp. 17–24. ACM (2007)Google Scholar
  3. 3.
    Bi, W., Kwok, J.T.: Multi-label classification on tree-and dag-structured hierarchies. In: Proceedings of the 28th International Conference on Machine Learning (ICML 2011), pp. 17–24 (2011)Google Scholar
  4. 4.
    Bi, W., Kwok, J.T.: Mandatory leaf node prediction in hierarchical multilabel classification. In: NIPS, pp. 153–161 (2012)Google Scholar
  5. 5.
    Bucak, S.S., Jin, R., Jain, A.K.: Multi-label learning with incomplete class assignments. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2801–2808. IEEE (2011)Google Scholar
  6. 6.
    Chen, X., Yuan, X.T., Chen, Q., Yan, S., Chua, T.S.: Multi-label visual classification with label exclusive context. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 834–841. IEEE (2011)Google Scholar
  7. 7.
    Cour, T., Sapp, B., Taskar, B.: Learning from partial labels. The Journal of Machine Learning Research 12, 1501–1536 (2011)zbMATHMathSciNetGoogle Scholar
  8. 8.
    Dekel, O., Keshet, J., Singer, Y.: Large margin hierarchical classification. In: Proceedings of the Twenty-first International Conference on Machine Learning, p. 27. ACM (2004)Google Scholar
  9. 9.
    Deng, J., Berg, A., Satheesh, S., Su, H., Khosla, A., Fei-Fei, L.: Imagenet large scale visual recognition challenge 2012 (2012), http://www.image-net.org/challenges/LSVRC/2012
  10. 10.
    Desai, C., Ramanan, D., Fowlkes, C.C.: Discriminative models for multi-class object layout. International Journal of Computer Vision 95(1), 1–12 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  11. 11.
    Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., Darrell, T.: Decaf: A deep convolutional activation feature for generic visual recognition. arXiv preprint arXiv:1310.1531 (2013)Google Scholar
  12. 12.
    Elhoseiny, M., Saleh, B., Elgammal, A.: Write a classifier: Zero-shot learning using purely textual descriptions. In: ICCV (2013)Google Scholar
  13. 13.
    Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. International Journal of Computer Vision 88(2), 303–338 (2010)CrossRefGoogle Scholar
  14. 14.
    Farhadi, A., Endres, I., Hoiem, D.: Attribute-centric recognition for cross-category generalization. In: 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2352–2359. IEEE (2010)Google Scholar
  15. 15.
    Fergus, R., Bernal, H., Weiss, Y., Torralba, A.: Semantic label sharing for learning with many categories. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part I. LNCS, vol. 6311, pp. 762–775. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  16. 16.
    Frome, A., Corrado, G.S., Shlens, J., Bengio, S., Dean, J., Mikolov, T.: Devise: A deep visual-semantic embedding model. In: Advances in Neural Information Processing Systems, pp. 2121–2129 (2013)Google Scholar
  17. 17.
    Hwang, S.J., Sha, F., Grauman, K.: Sharing features between objects and their attributes. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1761–1768. IEEE (2011)Google Scholar
  18. 18.
    Jia, Y., Abbott, J.T., Austerweil, J., Griffiths, T., Darrell, T.: Visual concept learning: Combining machine vision and bayesian generalization on concept hierarchies. In: Advances in Neural Information Processing Systems, pp. 1842–1850 (2013)Google Scholar
  19. 19.
    Jin, R., Ghahramani, Z.: Learning with multiple labels. In: Advances in Neural Information Processing Systems, pp. 897–904 (2002)Google Scholar
  20. 20.
    Kang, F., Jin, R., Sukthankar, R.: Correlated label propagation with application to multi-label learning. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 1719–1726. IEEE (2006)Google Scholar
  21. 21.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS, vol. 1, p. 4 (2012)Google Scholar
  22. 22.
    Kuettel, D., Guillaumin, M., Ferrari, V.: Segmentation propagation in imageNet. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part VII. LNCS, vol. 7578, pp. 459–473. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  23. 23.
    Lafferty, J.D., McCallum, A., Pereira, F.C.N.: Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In: Proceedings of the Eighteenth International Conference on Machine Learning, ICML 2001, pp. 282–289. Morgan Kaufmann Publishers Inc., San Francisco (2001), http://dl.acm.org/citation.cfm?id=645530.655813 Google Scholar
  24. 24.
    Lampert, C.H.: Maximum margin multi-label structured prediction. In: NIPS, vol. 11, pp. 289–297 (2011)Google Scholar
  25. 25.
    Lampert, C.H., Nickisch, H., Harmeling, S.: Learning to detect unseen object classes by between-class attribute transfer. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2009, pp. 951–958. IEEE (2009)Google Scholar
  26. 26.
    Lim, J.J., Salakhutdinov, R., Torralba, A.: Transfer learning by borrowing examples for multiclass object detection. In: Neural Information Processing Systems, NIPS (2011)Google Scholar
  27. 27.
    Marszalek, M., Schmid, C.: Semantic hierarchies for visual object recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2007, pp. 1–7. IEEE (2007)Google Scholar
  28. 28.
    Palatucci, M., Pomerleau, D., Hinton, G.E., Mitchell, T.M.: Zero-shot learning with semantic output codes. In: NIPS, vol. 3, pp. 5–2 (2009)Google Scholar
  29. 29.
    Perronnin, F., Akata, Z., Harchaoui, Z., Schmid, C.: Towards good practice in large-scale learning for image classification. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3482–3489. IEEE (2012)Google Scholar
  30. 30.
    Rohrbach, M., Stark, M., Schiele, B.: Evaluating knowledge transfer and zero-shot learning in a large-scale setting. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1641–1648. IEEE (2011)Google Scholar
  31. 31.
    Rohrbach, M., Stark, M., Szarvas, G., Gurevych, I., Schiele, B.: What helps where–and why? semantic relatedness for knowledge transfer. In: 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 910–917. IEEE (2010)Google Scholar
  32. 32.
    Sánchez, J., Perronnin, F., Mensink, T., Verbeek, J.: Image classification with the fisher vector: Theory and practice. International Journal of Computer Vision 105(3), 222–245 (2013)CrossRefzbMATHMathSciNetGoogle Scholar
  33. 33.
    Sharmanska, V., Quadrianto, N., Lampert, C.H.: Augmented attribute representations. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part V. LNCS, vol. 7576, pp. 242–255. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  34. 34.
    Tousch, A.M., Herbin, S., Audibert, J.Y.: Semantic hierarchies for image annotation: A survey. Pattern Recognition 45(1), 333–345 (2012)CrossRefGoogle Scholar
  35. 35.
    Williams, C., Seeger, M.: Using the nyström method to speed up kernel machines. In: Advances in Neural Information Processing Systems 13. Citeseer (2001)Google Scholar
  36. 36.
    Yu, F.X., Cao, L., Feris, R.S., Smith, J.R., Chang, S.F.: Designing category-level attributes for discriminative visual recognition. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 771–778. IEEE (2013)Google Scholar
  37. 37.
    Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional neural networks. arXiv preprint arXiv:1311.2901 (2013)Google Scholar
  38. 38.
    Zweig, A., Weinshall, D.: Exploiting object hierarchy: Combining models from different category levels. In: IEEE 11th International Conference on Computer Vision, ICCV 2007, pp. 1–8. IEEE (2007)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Jia Deng
    • 1
    • 2
  • Nan Ding
    • 2
  • Yangqing Jia
    • 2
  • Andrea Frome
    • 2
  • Kevin Murphy
    • 2
  • Samy Bengio
    • 2
  • Yuan Li
    • 2
  • Hartmut Neven
    • 2
  • Hartwig Adam
    • 2
  1. 1.University of MichiganUSA
  2. 2.Google Inc.USA

Personalised recommendations