Attributes Make Sense on Segmented Objects

  • Zhenyang Li
  • Efstratios Gavves
  • Thomas Mensink
  • Cees G. M. Snoek
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8694)


In this paper we aim for object classification and segmentation by attributes. Where existing work considers attributes either for the global image or for the parts of the object, we propose, as our first novelty, to learn and extract attributes on segments containing the entire object. Object-level attributes suffer less from accidental content around the object and accidental image conditions such as partial occlusions, scale changes and viewpoint changes. As our second novelty, we propose joint learning for simultaneous object classification and segment proposal ranking, solely on the basis of attributes. This naturally brings us to our third novelty: object-level attributes for zero-shot, where we use attribute descriptions of unseen classes for localizing their instances in new images and classifying them accordingly. Results on the Caltech UCSD Birds, Leeds Butterflies, and an a-Pascal subset demonstrate that i) extracting attributes on oracle object-level brings substantial benefits ii) our joint learning model leads to accurate attribute-based classification and segmentation, approaching the oracle results and iii) object-level attributes also allow for zero-shot classification and segmentation.We conclude that attributes make sense on segmented objects.


attributes segmentation zero-shot classification 


  1. 1.
    Akata, Z., Perronnin, F., Harchaoui, Z., Schmid, C.: Label-embedding for attribute-based classification. In: CVPR (2013)Google Scholar
  2. 2.
    Alexe, B., Deselaers, T., Ferrari, V.: Measuring the objectness of image windows. TPAMI (2012)Google Scholar
  3. 3.
    Arbelaez, P., Hariharan, B., Gu, C., Gupta, S., Bourdev, L., Malik, J.: Semantic segmentation using regions and parts. In: CVPR (2012)Google Scholar
  4. 4.
    Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: From contours to regions: An empirical evaluation. In: CVPR (2009)Google Scholar
  5. 5.
    Blaschko, M.B., Lampert, C.H.: Learning to localize objects with structured output regression. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part I. LNCS, vol. 5302, pp. 2–15. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  6. 6.
    Bourdev, L., Maji, S., Malik, J.: Describing people: A poselet-based approach to attribute classification. In: ICCV (2011)Google Scholar
  7. 7.
    Branson, S., Wah, C., Schroff, F., Babenko, B., Welinder, P., Perona, P., Belongie, S.: Visual recognition with humans in the loop. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part IV. LNCS, vol. 6314, pp. 438–451. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  8. 8.
    Carreira, J., Caseiro, R., Batista, J., Sminchisescu, C.: Semantic segmentation with second-order pooling. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part VII. LNCS, vol. 7578, pp. 430–443. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  9. 9.
    Carreira, J., Li, F., Sminchisescu, C.: Object recognition by sequential figure-ground ranking. IJCV (2012)Google Scholar
  10. 10.
    Carreira, J., Sminchisescu, C.: CPMC: Automatic object segmentation using constrained parametric min-cuts. TPAMI (2012)Google Scholar
  11. 11.
    Chai, Y., Lempitsky, V., Zisserman, A.: BiCoS: A bi-level co-segmentation method for image classification. In: ICCV (2011)Google Scholar
  12. 12.
    Chai, Y., Rahtu, E., Lempitsky, V., Van Gool, L., Zisserman, A.: TriCoS: A tri-level class-discriminative co-segmentation method for image classification. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part I. LNCS, vol. 7572, pp. 794–807. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  13. 13.
    Deng, J., Krause, J., Fei-Fei, L.: Fine-grained crowdsourcing for fine-grained recognition. In: CVPR (2013)Google Scholar
  14. 14.
    Duan, K., Parikh, D., Crandall, D., Grauman, K.: Discovering localized attributes for fine-grained recognition. In: CVPR (2012)Google Scholar
  15. 15.
    Endres, I., Hoiem, D.: Category-independent object proposals with diverse ranking. TPAMI (2014)Google Scholar
  16. 16.
    Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. IJCV (2010)Google Scholar
  17. 17.
    Farhadi, A., Endres, I., Hoiem, D., Forsyth, D.: Describing objects by their attributes. In: CVPR (2009)Google Scholar
  18. 18.
    Felzenszwalb, P., Girshick, R., McAllester, D., Ramanan, D.: Object detection with discriminatively trained part based models. TPAMI (2010)Google Scholar
  19. 19.
    Ferrari, V., Zisserman, A.: Learning visual attributes. In: NIPS (2007)Google Scholar
  20. 20.
    Gavves, E., Fernando, B., Snoek, C., Smeulders, A., Tuytelaars, T.: Fine-grained categorization by alignments. In: ICCV (2013)Google Scholar
  21. 21.
    Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: CVPR (2014)Google Scholar
  22. 22.
    Kolmogorov, V., Zabin, R.: What energy functions can be minimized via graph cuts? TPAMI (2004)Google Scholar
  23. 23.
    Kovashka, A., Grauman, K.: Attribute adaptation for personalized image search. In: ICCV (2013)Google Scholar
  24. 24.
    Lampert, C., Blaschko, M., Hofmann, T.: Efficient subwindow search: A branch and bound framework for object localization. TPAMI (2009)Google Scholar
  25. 25.
    Lampert, C., Nickisch, H., Harmeling, S.: Attribute-based transfer learning for object categorization with zero/one training example. TPAMI (2013)Google Scholar
  26. 26.
    Li, Z., Gavves, E., van de Sande, K., Snoek, C., Smeulders, A.: Codemaps segment, classify and search objects locally. In: ICCV (2013)Google Scholar
  27. 27.
    Manen, S., Guillaumin, M., Van Gool, L.: Prime object proposals with randomized prim’s algorithm. In: ICCV (2013)Google Scholar
  28. 28.
    Parikh, D., Grauman, K.: Relative attributes. In: ICCV (2011)Google Scholar
  29. 29.
    Sánchez, J., Perronnin, F., Mensink, T., Verbeek, J.: Image classification with the fisher vector: Theory and practice. IJCV (2013)Google Scholar
  30. 30.
    van de Sande, K., Gevers, T., Snoek, C.: Evaluating color descriptors for object and scene recognition. TPAMI (2010)Google Scholar
  31. 31.
    Tsochantaridis, I., Joachims, T., Hofmann, T., Altun, Y.: Large margin methods for structured and interdependent output variables (2005)Google Scholar
  32. 32.
    Uijlings, J., van de Sande, K., Gevers, T., Smeulders, A.: Selective search for object recognition. IJCV (2013)Google Scholar
  33. 33.
    Uijlings, J., Smeulders, A., Scha, R.: The visual extent of an object. IJCV (2012)Google Scholar
  34. 34.
    Usunier, N., Buffoni, D., Gallinar, P.: Ranking with ordered weighted pairwise classification. In: ICML (2009)Google Scholar
  35. 35.
    Vijayanarasimhan, S., Grauman, K.: Efficient region search for object detection. In: CVPR (2011)Google Scholar
  36. 36.
    Wah, C., Branson, S., Perona, P., Belongie, S.: Multiclass recognition and part localization with humans in the loop. In: ICCV (2011)Google Scholar
  37. 37.
    Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The Caltech-UCSD Birds-200-2011 Dataset. Tech. rep. (2011)Google Scholar
  38. 38.
    Wang, J., Markert, K., Everingham, M.: Learning models for object recognition from natural language descriptions. In: BMVC (2009)Google Scholar
  39. 39.
    Weston, J., Bengio, S., Usunier, N.: WSABIE: Scaling up to large vocabulary image annotation. In: IJCAI (2011)Google Scholar
  40. 40.
    Zhu, S., Angelova, A.: Efficient object detection and segmentation for fine-grained recognition. In: CVPR (2013)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Zhenyang Li
    • 1
  • Efstratios Gavves
    • 1
  • Thomas Mensink
    • 1
  • Cees G. M. Snoek
    • 1
  1. 1.ISLA, Informatics InstituteUniversity of AmsterdamThe Netherlands

Personalised recommendations