Advertisement

International Journal of Computer Vision

, Volume 111, Issue 2, pp 191–212 | Cite as

Local Alignments for Fine-Grained Categorization

  • Efstratios Gavves
  • Basura Fernando
  • Cees G. M. Snoek
  • Arnold W. M. Smeulders
  • Tinne Tuytelaars
Article

Abstract

The aim of this paper is fine-grained categorization without human interaction. Different from prior work, which relies on detectors for specific object parts, we propose to localize distinctive details by roughly aligning the objects using just the overall shape. Then, one may proceed to the classification by examining the corresponding regions of the alignments. More specifically, the alignments are used to transfer part annotations from training images to unseen images (supervised alignment), or to blindly yet consistently segment the object in a number of regions (unsupervised alignment). We further argue that for the distinction of sub-classes, distribution-based features like color Fisher vectors are better suited for describing localized appearance of fine-grained categories than popular matching oriented shape-sensitive features, like HOG. They allow capturing the subtle local differences between subclasses, while at the same time being robust to misalignments between distinctive details. We evaluate the local alignments on the CUB-2011 and on the Stanford Dogs datasets, composed of 200 and 120, visually very hard to distinguish bird and dog species. In our experiments we study and show the benefit of the color Fisher vector parameterization, the influence of the alignment partitioning, and the significance of object segmentation on fine-grained categorization. We, furthermore, show that by using object detectors as voters to generate object confidence saliency maps, we arrive at fully unsupervised, yet highly accurate fine-grained categorization. The proposed local alignments set a new state-of-the-art on both the fine-grained birds and dogs datasets, even without any human intervention. What is more, the local alignments reveal what appearance details are most decisive per fine-grained object category.

Keywords

Alignment Image representation  Object classification 

Notes

Acknowledgments

The projects IMPact BeeldCanon, AXES, STW STORY, COGNIMUND, and the Dutch national program COMMIT support this research.

References

  1. Alexe, B., Deselaers, T., & Ferrari, V. (2012). Measuring the objectness of image windows. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(11), 2189–2202.CrossRefGoogle Scholar
  2. Arbelaez, P. Hariharan, B. Gu, C. Gupta, S. Bourdev, L. & Malik, J. (2012). Semantic segmentation using regions and parts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012 (pp. 3378-3385). IEEE.Google Scholar
  3. Azizpour, H. & Laptev, I. (2012). Object detection using strongly-supervised deformable part models. In Proceedings of the European conference on Computer Vision, (pp. 836–849).Google Scholar
  4. Bay, H., Ess, A., Tuytelaars, T., & Van Gool, L. (2008). Speeded-up robust features (surf). Vision and Image Understanding: Computer, 110(3), 346–359.CrossRefGoogle Scholar
  5. Berg, T. & Belhumeur, P. N. (2013). POOF: Part-based one-vs.-one features for fine-grained categorization, face verification, and attribute estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (pp. 955–962). IEEE.Google Scholar
  6. Biederman, I. (1987). Recognition-by-components: A theory of human image understanding. Psychological Review, 94(2), 115.CrossRefGoogle Scholar
  7. Bo, L. Ren, X. & Fox, D. (2010) Kernel descriptors for visual recognition. In Proceedings of the Neural Information Processing Systems.Google Scholar
  8. Bourdev, L. & Malik, J. (2009). Poselets: Body part detectors trained using 3D human pose annotations. In IEEE International Conference on Computer Vision, (pp. 1365–1372). IEEE.Google Scholar
  9. Branson, S. Wah, C. Schroff, F. Babenko, B. Welinder, P. Perona, P. & Belongie, S. (2010). Visual recognition with humans in the loop. In Proceedings of the European Conference on Computer Vision.Google Scholar
  10. Branson, S. Perona, P. & Belongie, S. (2011). Strong supervision from weak annotation: Interactive training of deformable part models. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).Google Scholar
  11. Branson, S. Van Horn, G. Wah, C. Perona, P. & Belongie, S. (2014). The ignorant led by the blind: A hybrid human—machine vision system for fine-grained categorization. International Journal of Computer Vision, 1–27.Google Scholar
  12. Carreira, J. (2012). CPMC: Automatic object segmentation using constrained parametric min-cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(7), 1312–1328.CrossRefGoogle Scholar
  13. Chai, Y. Lempitsky, V. & Zisserman, A. (2011). BiCoS: A bi-level co-segmentation method for image classification. In IEEE International Conference on Computer Vision, (pp. 2579–2586). IEEE.Google Scholar
  14. Chai, Y. Rahtu, E. Lempitsky, V. Van Gool, L. & Zisserman, A. (2012). TriCoS: A tri-level class-discriminative co-segmentation method for image classification. In Proceedings of the European Conference on Computer Vision.Google Scholar
  15. Chai, Y. Lempitsky, V. & Zisserman, A. (2013). Symbiotic segmentation and part localization for fine-grained categorization. In IEEE International Conference on Computer Vision (ICCV). IEEE.Google Scholar
  16. Cinbis, R. G. Verbeek, J. & Schmid, C. (2013). Segmentation driven object detection with fisher vectors. In IEEE International Conference on Computer Vision.Google Scholar
  17. Dalal, N. & Triggs, B. (2005). Histograms of oriented gradients for human detection. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition.Google Scholar
  18. Darwin, C. (1859) On the Origin of Species by Means of Natural Selection, or the Preservation of Favoured Races in the Struggle for Life.Google Scholar
  19. Deng, J. Dong, W. Socher, R. Li, L.-J. Li, K. & Fei-Fei, L. (2009). ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  20. Donahue, J. Jia, Y. Vinyals, O. Hoffman, J. Zhang, N. Tzeng, E. & Darrell, T. (2013). DeCAF: A deep convolutional activation feature for generic visual recognition. Technical report. arXiv:1310.1531.
  21. Duan, K. Parikh, D. Crandall, D. & Grauman, K. (2012). Discovering localized attributes for fine-grained recognition. In Proceedings of the IEEE Conference on Vision and Pattern Recognition.Google Scholar
  22. Everingham, M. Van Gool, L. Williams, C. K. I. Winn, J. & Zisserman, A. (2007). The PASCAL Visual Object Classes Challenge (VOC2007) Results. http://www.pascal-network.org/challenges/VOC/voc2007 2007
  23. Farrell, R. Oza, O. Zhang, N. Morariu, V. I. Darrell, T. & Davis, L. S. (2011). Birdlets: Subordinate categorization using volumetric primitives and pose-normalized appearance. In Proceedings of the IEEE International Conference on Computer Vision.Google Scholar
  24. Felzenszwalb, P. F., & Huttenlocher, D. P. (2004). Efficient graph-based image segmentation. International Journal of Computer Vision, 59(2), 167–181.CrossRefGoogle Scholar
  25. Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2010). Object detection with discriminatively trained part-based models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(9), 1627–1645.CrossRefGoogle Scholar
  26. Gavves, E. Snoek, C. G. M. & Smeulders, A. W. M. (2012). Convex reduction of high-dimensional kernels for visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  27. Gavves, E. Fernando, B. Snoek, C. G. M. Smeulders, A. W. M. & Tuytelaars, T. (2013). Fine-grained categorization by alignments. In Proceedings of the IEEE International Conference on Computer Vision.Google Scholar
  28. Gosselin, P. H. Murray, N. Jégou, H. & Perronnin, F. (2013). Inria+Xerox@FGcomp: Boosting the fisher vector for fine-grained classification. Research Report RR-8431, INRIA.Google Scholar
  29. Itti, L., Koch, C., & Niebur, E. (1998). A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20, 1254–1259.CrossRefGoogle Scholar
  30. Jia, Y. Vinyals, O. & Darrell, T. (2013). Pooling-Invariant Image Feature Learning. Technical report. arXiv:1302.5056.
  31. Khosla, A. Jayadevaprakash, N. Yao, B. & Fei-Fei, L. (2011). Novel dataset for fine-grained image categorization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  32. Lazebnik, S. Schmid, C. & Ponce, J. (2006). Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition.Google Scholar
  33. Liu, J. Kanazawa, A. Jacobs, D. & Belhumeur, P. (2012). Dog breed classification using part localization. In Proceedings of the European Conference on Computer Vision.Google Scholar
  34. Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2), 91–110.CrossRefGoogle Scholar
  35. Maji, S. Berg, A. C. & Malik, J. (2008). Classification using intersection kernel support vector machines is efficient. In Proceedings of the IEEE Conference on Vision and Pattern Recognition.Google Scholar
  36. Maji, S. Kannala, J. Rahtu, E. Blaschko, M. & Vedaldi, A. (2013). Fine-grained visual classification of aircraft. Technical report.Google Scholar
  37. Manén, S. Guillaumin, M. & Van Gool, L. (2013). Prime object proposals with randomized prim’s algorithm. In Proceedings of the IEEE International Conference on Computer Vision.Google Scholar
  38. Mikolajczyk, K., Tuytelaars, T., Schmid, C., Zisserman, A., Matas, J., Schaffalitzky, F., et al. (2005). A comparison of affine region detectors. International Journal of Computer Vision, 65, 43–72.CrossRefGoogle Scholar
  39. Nilsback, M.E. & Zisserman, A. (2008). Automated flower classification over a large number of classes. In ICVGIP.Google Scholar
  40. Parikh, D. & Grauman, K. (2011). Interactive discovery of task-specific nameable attributes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  41. Parkhi, O. M. Vedaldi, A. Zisserman, A. & Jawahar, C. V. (2012). Cats and dogs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  42. Perdóch, M. Chum, O. & Matas, J. (2009). Efficient representation of local geometry for large scale object retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  43. Perronnin, F. Sanchez, J. & Mensink, T. (2010). Improving the fisher kernel for large-scale image classification. In Procedings of the European Conference Computer Vision.Google Scholar
  44. Rosch, E., Mervis, C. B., Gray, W. D., Johnson, D. M., & Boyes-Braem, P. (1976). Basic objects in natural categories. Cognitive Psychology, 8, 382–439.CrossRefGoogle Scholar
  45. Rother, C. Kolmogorov, V. & Blake, A. (2004). Interactive foreground extraction using iterated graph cuts. In ACM Transactions on Graphics: Grabcut. ACMGoogle Scholar
  46. Sanchez, J. Perronnin, F. & Akata, Z. (2011). Fisher vectors for fine-grained visual categorization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition.Google Scholar
  47. Shalev-Shwartz, S. Singer, Y. & Srebro, N. (2007) Pegasos: Primal estimated sub-gradient solver for svm. In Proceedings of the International Conference on Machine Learning.Google Scholar
  48. Swain, M. J., & Ballard, D. H. (1991). Color indexing. International Journal of Computer Vision, 7, 11–32.CrossRefGoogle Scholar
  49. Uijlings, J. R. R., van de Sande, K. E. A., Gevers, T., & Smeulders, A. W. M. (2013). Selective search for object recognition. International Journal of Computer Vision, 104, 154–171.CrossRefGoogle Scholar
  50. van de Sande, K. E. A. Gevers, T. & Snoek, C. G. M. (2010). Evaluating color descriptors for object and scene recognition. In IEEE Transactions on Pattern Analysis and Machine Intelligence.Google Scholar
  51. Vedaldi, A. & Fulkerson, B. (2010). VLFeat: An open and portable library of computer vision algorithms. In Proceedings of the International Conference on Multimedia. ACMGoogle Scholar
  52. Vedaldi, A. Gulshan, V. Varma, M. & Zisserman, A. (2009). Multiple kernels for object detection. In Proceedings of the International Conference on Vision.Google Scholar
  53. Wah, C. Branson, S. Perona, P. & Belongie, S. (2011a). Multiclass recognition and part localization with humans in the loop. In Proceedings of the IEEE International Conference on Computer Vision. Google Scholar
  54. Wah, C. Branson, S. Welinder, P. Perona, P. & Belongie, S. (2011b). The Caltech-UCSD Birds-200-2011 Dataset. Technical report.Google Scholar
  55. Xie, L. Tian, Q. Yan, B. & Zhang, S. (2013). Hierarcical part matching for fine-grained visual categorization. In Proceedings of the IEEE Conference on Computer Vision.Google Scholar
  56. Yang, S. Bo, L. Wang, J. & Shapiro, L. (2012). Unsupervised template learning for fine-grained object recognition. In Proceedings of the Neural Information Processing Systems.Google Scholar
  57. Yao, B. Khosla, A. & Fei-Fei, L. (2011). Combining randomization and discrimination for fine-grained image categorization. In Proceedings of the IEEE Conference on Vision and Pattern Recognition.Google Scholar
  58. Yao, B. Bradski, G. & Fei-Fei, L. (2012). A codebook-free and annotation-free approach for fine-grained image categorization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  59. Zhang, N. Farrell, R. & Darrell, T. (2012). Pose pooling kernels for sub-category recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  60. Zhang, N. Farrell, R. Iandola, F. & Darrell, T. (2013). Deformable part descriptors for fine-grained recognition and attribute prediction. In Proceedings of the IEEE Conference on Computer Vision.Google Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Efstratios Gavves
    • 1
  • Basura Fernando
    • 2
  • Cees G. M. Snoek
    • 1
  • Arnold W. M. Smeulders
    • 1
  • Tinne Tuytelaars
    • 2
  1. 1.University of AmsterdamAmsterdamNetherlands
  2. 2. KU Leuven, ESAT PSI, iMindsLeuvenBelgium

Personalised recommendations