Visual Classification by a Hierarchy of Extended Fragments

  • Shimon Ullman
  • Boris Epshtein
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4170)


The chapter describes visual classification by a hierarchy of semantic fragments. In fragment-based classification, objects within a class are represented by common sub-structures selected during training. The chapter describes two extensions to the basic fragment-based scheme. The first extension is the extraction and use of feature hierarchies. We describe a method that automatically constructs complete feature hierarchies from image examples, and show that features constructed hierarchically are significantly more informative and better for classification compared with similar non-hierarchical features. The second extension is the use of so-called semantic fragments to represent object parts. The goal of a semantic fragment is to represent the different possible appearances of a given object part. The visual appearance of such object parts can differ substantially, and therefore traditional image similarity-based methods are inappropriate for the task. We show how the method can automatically learn the part structure of a new domain, identify the main parts, and how their appearance changes across objects in the class. We discuss the implications of these extensions to object classification and recognition.


Mutual Information Image Patch Visual Similarity Object Part Root Fragment 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Agarwal, S., Awan, A., Roth, D.: Learning to detect objects in images via a sparse, part-based representation. IEEE TPAMI 26(11), 1475–1490 (2004)Google Scholar
  2. 2.
    Bart, E., Ullman, S.: Class-based matching of object parts. In: Proc. CVPR Workshop on Image and Video Registration (2004)Google Scholar
  3. 3.
    Biederman, I.: Recognition-by-Components: A Theory of Human Image Understanding. Psychological Review 94(2), 115–147 (1987)CrossRefGoogle Scholar
  4. 4.
    Epshtein, B., Ullman, S.: Identifying Semantically Equivalent Object Fragments. In: CVPR, pp. 2–9 (2005)Google Scholar
  5. 5.
    Epshtein, B., Ullman, S.: Feature Hierarchies for Object Classification. In: ICCV (to appear, 2005)Google Scholar
  6. 6.
    Fergus, R., Perona, P., Zisserman, A.: Object Class Recognition by Unsupervised Scale-Invariant Learning. In: CVPR, pp. 264–271 (2003)Google Scholar
  7. 7.
    Foldiak, P.: Learning invariance from transformation sequences. Neural Computation 3(2), 194–200 (1991)CrossRefGoogle Scholar
  8. 8.
    Green, D., Swets, J.: Signal Detection Theory and Psychophysics. Wiley, NY (1966)Google Scholar
  9. 9.
    Heisele, B., Serre, T., Pontil, M., Vetter, T., Poggio, T.: Categorization by learning and combining object parts. In: NIPS (2001)Google Scholar
  10. 10.
    Itti, L., Kosh, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE TPAMI 20(11), 1254–1259 (1998)Google Scholar
  11. 11.
    LeCun, Y., Boser, B., Denker, J., Henderson, D., Howard, R., Hubbard, W., Jackel, L.: Backpropagation applied to handwritten zip code recognition. Neural Computation 1(4), 541–551 (1989)CrossRefGoogle Scholar
  12. 12.
    Lowe, D.: Distinctive image features from scale-invariant keypoints. Int. J. Comp. Vis. 60(2), 91–100 (2004)CrossRefGoogle Scholar
  13. 13.
    Marr, D., Nishihara, H.: Representation and recognition of the spatial organization of three dimensional structure. Proceedings of the Royal Society of London B 200, 269–294 (1978)CrossRefGoogle Scholar
  14. 14.
    Mikolajczyk, K., Schmidt, C.: A performance evaluation of local descriptors. In: CVPR, pp. 257–264 (2003)Google Scholar
  15. 15.
    Mikolajczyk, K., Schmidt, C.: Scale and affine invariant point detectors. Int. J. Comp. Vis. 60(1), 63–86 (2004)CrossRefGoogle Scholar
  16. 16.
    Riesenhuber, M., Poggio, T.: Hierarchical models of object recognition in cortex. Nature Neuroscience 2(11), 1019–1025 (1999)CrossRefGoogle Scholar
  17. 17.
    Stringer, S., Rolls, E.: Invariant object recognition in the visual system with novel view of 3D objects. Neural Computation 14, 2585–2596 (2002)MATHCrossRefGoogle Scholar
  18. 18.
    Tomasi, C., Kanade, T.: Detecting and tracking of point features. Technical Report CMU-CS-91-132, Carnegie Mellon University (1991)Google Scholar
  19. 19.
    Ullman, S., Bart, E.: Recognition invariance obtained by extended and invariant features. Neural Networks 17, 833–848 (2004)MATHCrossRefGoogle Scholar
  20. 20.
    Ullman, S., Soloviev, S.: Computation of pattern invariance in brain-like structures. Neural Networks 12, 1021–1036 (1999)CrossRefGoogle Scholar
  21. 21.
    Ullman, S., Vidal-Naquet, M., Sali, E.: Visual features of intermediate complexity and their use in classification. Nature Neuroscience 5(7), 1–6 (2002)Google Scholar
  22. 22.
    Vidal-Naquet, M., Ullman, S.: Object Recognition with Informative Features and Linear Classification. In: ICCV, pp. 281–288 (2003)Google Scholar
  23. 23.
    Wiskott, L., Fellous, J., Kruger, N., von der Malsburg, C.: Face Recognition by Elastic Bunch Graph Matching. IEEE TPAMI 19(7), 775–779 (1997)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Shimon Ullman
    • 1
  • Boris Epshtein
    • 1
  1. 1.Weizmann Institute of ScienceRehovotIsrael

Personalised recommendations