Evaluation of Image Segmentation Quality by Adaptive Ground Truth Composition

  • Bo Peng
  • Lei Zhang
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7574)


Segmenting an image is an important step in many computer vision applications. However, image segmentation evaluation is far from being well-studied in contrast to the extensive studies on image segmentation algorithms. In this paper, we propose a framework to quantitatively evaluate the quality of a given segmentation with multiple ground truth segmentations. Instead of comparing directly the given segmentation to the ground truths, we assume that if a segmentation is “good”, it can be constructed by pieces of the ground truth segmentations. Then for a given segmentation, we construct adaptively a new ground truth which can be locally matched to the segmentation as much as possible and preserve the structural consistency in the ground truths. The quality of the segmentation can then be evaluated by measuring its distance to the adaptively composite ground truth. To the best of our knowledge, this is the first work that provides a framework to adaptively combine multiple ground truths for quantitative segmentation evaluation. Experiments are conducted on the benchmark Berkeley segmentation database, and the results show that the proposed method can faithfully reflect the perceptual qualities of segmentations.


Image segmentation evaluation ground truths image segmentation 


  1. 1.
    Liu, J., Yang, Y.: Multiresolution color image segmentation. IEEE Trans. on Pattern Analysis and Machine Intelligence 16, 689–700 (1994)CrossRefGoogle Scholar
  2. 2.
    Borsotti, M., Campadelli, P., Schettini, R.: Quantitative evaluation of color image segmenta- tion results. Pattern Recognition Letter 19, 741–747 (1998)zbMATHCrossRefGoogle Scholar
  3. 3.
    Zhang, H., Fritts, J., Goldman, S.: An entropy-based objective segmentation eval- uation method for image segmentation. In: SPIE Electronic Imaging Storage and Retrieval Methods and Applications for Multimedia, pp. 38–49 (2004)Google Scholar
  4. 4.
    Ren, X., Jitendra, M.: Learning a classification model for segmentation. In: IEEE Conference on Computer Vision, pp. 10–17 (2003)Google Scholar
  5. 5.
    Christensen, H., Phillips, P.: Empirical evaluation methods in computer vision. World Scientific Publishing Company (2002)Google Scholar
  6. 6.
    Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: International Conference on Computer Vision, pp. 416–423 (2001)Google Scholar
  7. 7.
    Freixenet, J., Muñoz, X., Raba, D., Martí, J., Cufí, X.: Yet Another Survey on Image Segmentation: Region and Boundary Information Integration. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002, Part III. LNCS, vol. 2352, pp. 408–422. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  8. 8.
    Martin, D.: An empirical approach to grouping and segmentation. Ph.D. dissertation U.C. Berkeley (2002)Google Scholar
  9. 9.
    Rand, W.: Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association 66, 846–850 (1971)CrossRefGoogle Scholar
  10. 10.
    Fowlkes, E., Mallows, C.: A method for comparing two hierarchical clusterings. Journal of the American Statistical Association 78, 553–569 (1983)zbMATHCrossRefGoogle Scholar
  11. 11.
    Unnikrishnan, R., Hebert, M.: Measures of similarity. In: IEEE Workshop on Applications of Computer Vision, pp. 394–400 (2005)Google Scholar
  12. 12.
    Felzenszwalb, P., Huttenlocher, D.: Efficient graph-based image segmentation. International Journal on Computer Vision 59, 167–181 (2004)CrossRefGoogle Scholar
  13. 13.
    Viola, P., Wells, W.: Alignment by maximization of mutual information, vol. 3, pp. 16–23 (1995)Google Scholar
  14. 14.
    Wang, Z., Bovik, A.: Modern image quality assessment. Morgan and Claypool Publishing Company, New York (2006)Google Scholar
  15. 15.
    Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Process. 13, 600–612 (2004)CrossRefGoogle Scholar
  16. 16.
    Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., Cohen, M.: Interactive digital photomontage, vol. 23, pp. 294–302 (2004)Google Scholar
  17. 17.
    Russell, B., Efros, A., Sivic, J., Freeman, W., Zisserman, A.: Segmenting scenes by matching image composites. In: Advances in Neural Information Processing Systems, pp. 1580–1588 (2009)Google Scholar
  18. 18.
    Potts, R.: Some generalized order-disorder transformation. Proceedings of the Cambridge Philosophical Society 48, 106–109 (1952)MathSciNetzbMATHCrossRefGoogle Scholar
  19. 19.
    Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 1222–1239 (2001)CrossRefGoogle Scholar
  20. 20.
    Martin, D., Fowlkes, C., Malik, J.: Learning to detect natural image boundaries using local brightness, color and texture cues. IEEE Trans. on Pattern Analysis and Machine Intelligence 26, 530–549 (2004)CrossRefGoogle Scholar
  21. 21.
    Sampat, M., Wang, Z., Gupta, S., Bovik, A., Markey, M.: Complex wavelet structural similarity: A new image similarity index. IEEE Transactions on Image Processing 18, 2385–2401 (2009)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Comanicu, D., Meer, P.: Mean shift: A robust approach toward feature space analysis. IEEE Trans. on Pattern Analysis and Machine Intelligence 24, 603–619 (2002)CrossRefGoogle Scholar
  23. 23.
    Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 888–905 (1997)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Bo Peng
    • 1
    • 2
  • Lei Zhang
    • 2
  1. 1.Dept. of Software EngineeringSouthwest Jiaotong UniversityChina
  2. 2.Dept. of ComputingThe Hong Kong Polytechnic UniversityHong Kong

Personalised recommendations