Object Recognition by Integrating Multiple Image Segmentations

  • Caroline Pantofaru
  • Cordelia Schmid
  • Martial Hebert
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5304)


The joint tasks of object recognition and object segmentation from a single image are complex in their requirement of not only correct classification, but also deciding exactly which pixels belong to the object. Exploring all possible pixel subsets is prohibitively expensive, leading to recent approaches which use unsupervised image segmentation to reduce the size of the configuration space. Image segmentation, however, is known to be unstable, strongly affected by small image perturbations, feature choices, or different segmentation algorithms. This instability has led to advocacy for using multiple segmentations of an image. In this paper, we explore the question of how to best integrate the information from multiple bottom-up segmentations of an image to improve object recognition robustness. By integrating the image partition hypotheses in an intuitive combined top-down and bottom-up recognition approach, we improve object and feature support. We further explore possible extensions of our method and whether they provide improved performance. Results are presented on the MSRC 21-class data set and the Pascal VOC2007 object segmentation challenge.


Ground Truth Image Segmentation Object Recognition Object Segmentation Ground Truth Label 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Unnikrishnan, R., Pantofaru, C., Hebert, M.: Toward objective evaluation of image segmentation algorithms. PAMI 29 (2007)Google Scholar
  2. 2.
    Malisiewicz, T., Efros, A.A.: Improving spatial support for objects via multiple segmentations. In: BMVC (2007)Google Scholar
  3. 3.
    Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: ICCV (2001)Google Scholar
  4. 4.
    Russell, B., Efros, A., Sivic, J., Freeman, W., Zisserman, A.: Using multiple segmentations to discover objects and their extent in image collections. In: CVPR (2006)Google Scholar
  5. 5.
    Pantofaru, C., Dorkó, G., Schmid, C., Hebert, M.: Combining regions and patches for object class localization. In: Beyond Patches Workshop, CVPR (2006)Google Scholar
  6. 6.
    Hoiem, D., Efros, A., Hebert, M.: Recovering surface layout from an image. IJCV 75 (2007)Google Scholar
  7. 7.
    Azran, A., Ghahramani, Z.: Spectral methods for automatic multiscale data clustering. In: CVPR (2006)Google Scholar
  8. 8.
    Tu, Z., Chen, Z., Yuille, A.L., Zhu, S.C.: Image parsing: Unifying segmentation, detection, and recognition. IJCV (2005)Google Scholar
  9. 9.
    Borenstein, E., Malik, J.: Shape guided object segmentation. In: CVPR (2006)Google Scholar
  10. 10.
    Comaniciu, D., Meer, P.: Mean shift: A robust approach toward feature space analysis. PAMI (2002)Google Scholar
  11. 11.
    Fowlkes, C., Martin, D., Malik, J.: Learning affinity functions for image segmentation: Combining patch-based and gradient-based approaches. In: CVPR (2003)Google Scholar
  12. 12.
    Martin, D., Fowlkes, C., Malik, J.: Learning to detect natural image boundaries using local brightness, color and texture cues. PAMI (2003)Google Scholar
  13. 13.
    Felzenszwalb, P., Huttenlocher, D.: Efficient graph-based image segmentation. IJCV 59, 167–181 (2004)CrossRefGoogle Scholar
  14. 14.
    Shotton, J., Winn, J., Rother, C., Criminisi, A.: Textonboost: Joint appearance, shape and context modeling for multi-class object recognition and segmentation. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951. Springer, Heidelberg (2006)Google Scholar
  15. 15.
    Winn, J., Jojic, N.: Locus: Learning object classes with unsupervised segmentation. In: ICCV (2005)Google Scholar
  16. 16.
    Tu, Z., Zhu, S.C.: Image segmentation by data-driven markov chain monte carlo. PAMI 24, 657–673 (2002)CrossRefGoogle Scholar
  17. 17.
    Verbeek, J., Triggs, B.: Region classification with markov field aspect models. In: CVPR (2007)Google Scholar
  18. 18.
    Winn, J., Shotton, J.: The layout consistent random field for recognizing and segmenting partially occluded objects. In: CVPR (2006)Google Scholar
  19. 19.
    Kumar, M., Torr, P., Zisserman, A.: Obj cut. In: CVPR (2005)Google Scholar
  20. 20.
    Leibe, B., Schiele, B.: Interleaved object categorization and segmentation. In: BMVC (2003)Google Scholar
  21. 21.
    Shotton, J., Winn, J., Rother, C., Criminisi, A.: The MSRC 21-class object recognition database (2006)Google Scholar
  22. 22.
    Everingham, M., Van Gool, L., Williams, C., Winn, J., Zisserman, A.: The PASCAL VOC 2007 (2007),
  23. 23.
    Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines, Software (2001),
  24. 24.
    Wu, T.F., Lin, C.J., Weng, R.C.: Probability estimates for multi-class classification by pairwise coupling. Journal of Machine Learning Research 5, 975–1005 (2004)MathSciNetzbMATHGoogle Scholar
  25. 25.
    van de Weijer, J., Schmid, C.: Coloring local feature extraction. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951. Springer, Heidelberg (2006)Google Scholar
  26. 26.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. IJCV 60, 91–110 (2004)CrossRefGoogle Scholar
  27. 27.
    Ladicky, L., Kohli, P., Torr, P.: Oxford Brookes entry, PASCAL VOC 2007 Segmentation Challenge (2007),
  28. 28.
    Viitaniemi, V.: Helsinki University of Technology, PASCAL VOC 2007 Challenge (2007),
  29. 29.
    Ren, X., Malik, J.: Learning a classification model for segmentation. In: ICCV (2003)Google Scholar
  30. 30.
    Friedman, J., Hastie, T., Tibshirani, R.: Additive logistic regression: a statistical view of boosting. Annals of Statistics (2000)Google Scholar
  31. 31.
    Collins, M., Schapire, R., Singer, Y.: Logistic regression, Adaboost and Bregman distances. Machine Learning (2002)Google Scholar
  32. 32.
    Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. PAMI 23, 1222–1239 (2001)CrossRefGoogle Scholar
  33. 33.
    Russell, B., Torralba, A., Murphy, K., Freeman, W.T.: Labelme: a database and web-based tool for image annotation. IJCV (2007)Google Scholar
  34. 34.
    von Ahn, L., Liu, R., Blum, M.: Peekaboom: A game for locating objects in images. In: ACM CHI (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Caroline Pantofaru
    • 1
  • Cordelia Schmid
    • 2
  • Martial Hebert
    • 1
  1. 1.The Robotics InstituteCarnegie Mellon UniversityUSA
  2. 2.INRIA Grenoble, LEAR, LJKFrance

Personalised recommendations