Matching with Shape Contexts

  • Serge Belongie
  • Greg Mori
  • Jitendra Malik
Part of the Modeling and Simulation in Science, Engineering and Technology book series (MSSET)


We present a novel approach to measuring similarity between shapes and exploit it for object recognition. In our framework, the measurement of similarity is preceded by (1) solving for correspondences between points on the two shapes, and (2) using the correspondences to estimate an aligning transform. In order to solve the correspondence problem, we attach a descriptor, the shape context, to each point. The shape context at a reference point captures the distribution of the remaining points relative to it, thus offering a globally discriminative characterization. Corresponding points on two similar shapes will have similar shape contexts, enabling us to solve for correspondences as an optimal assignment problem. Given the point correspondences, we estimate the transformation that best aligns the two shapes; regularized thin-plate splines provide a flexible class of transformation maps for this purpose. The dissimilarity between the two shapes is computed as a sum of matching errors between corresponding points, together with a term measuring the magnitude of the aligning transform. We treat recognition in a nearest neighbor classification framework as the problem of finding the stored prototype shape that is maximally similar to that in the image. We also demonstrate that shape contexts can be used to quickly prune a search for similar shapes. We present two algorithms for rapid shape retrieval: representative shape contexts, performing comparisons based on a small number of shape contexts, and shapemes, using vector quantization in the space of shape contexts to obtain prototypical shape pieces. Results are presented for silhouettes, handwritten digits and visual CAPTCHAs.

Key words

Shape distance shape correspondence thin-plate spline (TPS) object recognition 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Y. Amit, D. Geman, and K. Wilder. Joint induction of shape features and tree classifiers. IEEE Trans. Pattern Analysis and Machine Intelligence, 19(11):1300–1305, November 1997.Google Scholar
  2. 2.
    S. Belongie, J. Malik, and J. Puzicha. Matching shapes. In Proc. 8th Int. Conf. Computer Vision, volume 1, pages 454–461, July 2001.Google Scholar
  3. 3.
    S. Belongie, J. Malik, and J. Puzicha. Shape context: A new descriptor for shape matching and object recognition. In T. K. Leen, T. G. Dietterich, and V. Tresp, editors, Advances in Neural Information Processing Systems 13: Proceedings of the 2000 Conference, pages 831–837, 2001.Google Scholar
  4. 4.
    S. Belongie, J. Malik, and J. Puzicha. Shape matching and object recognition using shape contexts. IEEE Trans. Pattern Analysis and Machine Intelligence, 24(4):509–522, April 2002.Google Scholar
  5. 5.
    A. Berg, T. Berg, and J. Malik. Shape matching and object recognition using low distortion correspondences. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 26–33, San Diego, CA, June 2005.Google Scholar
  6. 6.
    A. Berg and J. Malik. Geometric blur for template matching. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 607–614, Kauai, HI, December 2001.Google Scholar
  7. 7.
    P. J. Bickel. A distribution free version of the Smirnov two-sample test in the multivariate case. Annals of Mathematical Statistics, 40:1–23, 1969.CrossRefzbMATHGoogle Scholar
  8. 8.
    F. L. Bookstein. Principal warps: thin-plate splines and decomposition of deformations. IEEE Trans. Pattern Analysis and Machine Intelligence, 11(6):567–585, June 1989.Google Scholar
  9. 9.
    F. L. Bookstein. Morphometric tools for landmark data: geometry and biology. Cambridge Univ. Press, London, 1991.zbMATHGoogle Scholar
  10. 10.
    H. Chui and A. Rangarajan. A new algorithm for non-rigid point matching. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 44–51, Hilton Head, SC, June 2000.Google Scholar
  11. 11.
    D. DeCoste and B. Schölkopf. Training invariant support vector machines. Machine Learning, 46(1–3):161–190, January–February–March 2002.Google Scholar
  12. 12.
    G. Dorko and C. Schmid. Selection of scale invariant neighborhoods for object class recognition. In Proc. 9th Int. Conf. Computer Vision, pages 634–640, 2003.Google Scholar
  13. 13.
    J. Duchon. Splines minimizing rotation-invariant semi-norms in Sobolev spaces. In W. Schempp and K. Zeller, editors, Constructive Theory of Functions of Several Variables, pages 85–100. Springer-Verlag, Berlin, 1977.CrossRefGoogle Scholar
  14. 14.
    R. Fergus, P. Perona, and A. Zisserman. Object class recognition by unsupervised scale-invariant learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, volume 2, pages 264–271, Madison, WI, June 2003.Google Scholar
  15. 15.
    M. Fischler and R. Elschlager. The representation and matching of pictorial structures. IEEE Trans. Computers, C-22(1):67–92, 1973.CrossRefGoogle Scholar
  16. 16.
    A. Frome, D. Huber, R. Kolluri, T. Bulow, and J. Malik. Recognizing objects in range data using regional point descriptors. In Proc. 8th Europ. Conf. Comput. Vision, volume 3, pages 224–237, 2004.Google Scholar
  17. 17.
    F. Girosi, M. Jones, and T. Poggio. Regularization theory and neural networks architectures. Neural Computation, 7(2):219–269, 1995.Google Scholar
  18. 18.
    U. Grenander, Y. Chow, and D. Keenan. HANDS: A Pattern Theoretic Study of Biological Shapes. Springer, New York, 1991.Google Scholar
  19. 19.
    H. Guo, A. Rangarajan, S. Joshi, and L. Younes. A new joint clustering and diffeomorphism estimation algorithm for non-rigid shape matching. In IEEE Workshop on Articulated and Non-rigid motion (ANM), Washington, DC, 2004.Google Scholar
  20. 20.
    A. Ben Hamza and H. Krim. Geodesic object representation and recognition. In Discrete Geometry for Computer Imagery: 11th International Conference, DGCI, pages 378–387, 2003.Google Scholar
  21. 21.
    S. Jeannin and M. Bober. Description of core experiments for MPEG-7 motion/shape. Technical Report ISO/IEC JTC 1/SC 29/WG 11 MPEG99/N2690, MPEG-7, Seoul, March 1999.Google Scholar
  22. 22.
    A. E. Johnson and M. Hebert. Recognizing objects by matching oriented points. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 684–689, San Juan, Puerto Rico, 1997.Google Scholar
  23. 23.
    D. Jones and J. Malik. Computational framework to determining stereo correspondence from a set of linear spatial filters. Image and Vision Computing, 10(10):699–708, Dec. 1992.Google Scholar
  24. 24.
    R. Jonker and A. Volgenant. A shortest augmenting path algorithm for dense and sparse linear assignment problems. Computing, 38:325–340, 1987.zbMATHCrossRefMathSciNetGoogle Scholar
  25. 25.
    M. Lades, C. Vorbrüggen, J. Buhmann, J. Lange, C. von der Malsburg, R. Wurtz, and W. Konen. Distortion invariant object recognition in the dynamic link architecture. IEEE Trans. Computers, 42(3):300–311, March 1993.Google Scholar
  26. 26.
    L. J. Latecki, R. Lakämper, and U. Eckhardt. Shape descriptors for non-rigid shapes with a single closed contour. In Proc. IEEE Conf. Comput. Vision and Pattern Recognition, pages 424–429, 2000.Google Scholar
  27. 27.
    Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, November 1998.Google Scholar
  28. 28.
    D. G. Lowe. Distinctive image features from scale-invariant keypoints. Int. Journal of Computer Vision, 60(2):91–110, 2004.CrossRefGoogle Scholar
  29. 29.
    D. Martin, C. Fowlkes, and J. Malik. Learning to find brightness and texture boundaries in natural images. NIPS, 2002.Google Scholar
  30. 30.
    J. Meinguet. Multivariate interpolation at arbitrary points made simple. J. Appl. Math. Phys. (ZAMP), 5:439–468, 1979.Google Scholar
  31. 31.
    G. Mori, S. Belongie, and J. Malik. Shape contexts enable efficient retrieval of similar shapes. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., Kauai, HI, December 2001.Google Scholar
  32. 32.
    G. Mori and J. Malik. Recognizing objects in adversarial clutter: Breaking a visual captcha. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., volume 1, pages 134–141, Madison, WI, 2003.Google Scholar
  33. 33.
    R. Osada, T. Funkhouser, B. Chazelle, and D. Dobkin. Shape distributions. A CM Transactions on Graphics, 21(4):807–832, October 2002.Google Scholar
  34. 34.
    C. Papadimitriou and K. Stieglitz. Combinatorial Optimization: Algorithms and Complexity. Prentice-Hall, Upper Saddle River, NJ, 1982.zbMATHGoogle Scholar
  35. 35.
    M. J. D. Powell. A thin plate spline method for mapping curves into curves in two dimensions. In Computational Techniques and Applications (CTAC95), Melbourne, Australia, 1995.Google Scholar
  36. 36.
    B. D. Ripley. Pattern Recognition and Neural Networks. Cambridge University Press, Cambridge, 1996.zbMATHGoogle Scholar
  37. 37.
    E. Rosch. Natural categories. Cognitive Psychology, 4(3):328–350, 1973.CrossRefGoogle Scholar
  38. 38.
    E. Rosch, C. B. Mervis, W. D. Gray, D. M. Johnson, and P. Boyes-Braem. Basic objects in natural categories. Cognitive Psychology, 8(3):382–439, 1976.CrossRefGoogle Scholar
  39. 39.
    J. G. Snodgrass and M. Vanderwart. A standardized set of 260 pictures: Norms for name agreement, familiarity and visual complexity. Journal of Experimental Psychology: Human Learning and Memory, 6:174–215, 1980.CrossRefGoogle Scholar
  40. 40.
    A. Thayananthan, B. Stenger, P. H. S. Torr, and R. Cipolla. Shape context and chamfer matching in cluttered scenes. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., volume I, pages 127–133, Madison, WI, June 2003.Google Scholar
  41. 41.
    D’Arcy Wentworth Thompson. On Growth and Form. Cambridge University Press, London, 1917.Google Scholar
  42. 42.
    L. von Ahn, M. Blum, and J. Langford. Telling humans and computers apart (automatically). CMU Tech Report CMU-CS-02-117, February 2002.Google Scholar
  43. 43.
    G. Wahba. Spline Models for Observational Data. SIAM, 1990.Google Scholar
  44. 44.
    A. Yuille. Deformable templates for face recognition. J. Cognitive Neuroscience, 3(1):59–71, 1991.CrossRefGoogle Scholar

Copyright information

© Birkhäuser Boston 2006

Authors and Affiliations

  • Serge Belongie
    • 1
  • Greg Mori
    • 2
  • Jitendra Malik
    • 3
  1. 1.University of CaliforniaSan Diego, La JollaUSA
  2. 2.Simon Fraser UniversityBurnabyCanada
  3. 3.University of CaliforniaBerkeley, BerkeleyUSA

Personalised recommendations