Evaluating Image Segmentation Algorithms Using the Pareto Front

  • Mark Everingham
  • Henk Muller
  • Barry Thomas
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2353)


Image segmentation is the first stage of processing in many practical computer vision systems. While development of particular segmentation algorithms has attracted considerable research interest, relatively little work has been published on the subject of their evaluation. In this paper we propose the use of the Pareto front to allow evaluation and comparison of image segmentation algorithms in multi-dimensional fitness spaces, in a manner somewhat analogous to the use of receiver operating characteristic curves in binary classification problems. The principle advantage of this approach is that it avoids the need to aggregate metrics capturing multiple objectives into a single metric, and thus allows trade-offs between multiple aspects of algorithm behavior to be assessed. This is in contrast to previous approaches which have tended to use a single measure of “goodness”, or discrepancy to ground truth data. We define the Pareto front in the context of algorithm evaluation, propose several fitness measures for image segmentation, and use a genetic algorithm for multi-objective optimization to explore the set of algorithms, parameters, and corresponding points in fitness space which lie on the front. Experimental results are presented for six general-purpose image segmentation algorithms, including several which may be considered state-of-the-art.


Vision systems engineering & evaluation Image segmentation Multiobjective evaluation Pareto front 


  1. 1.
    Kim, I.Y., Yang, H.S.: An integrated approach for scene understanding based on Markov random field model. Pattern Recognition 28 (1995) 1887–1897CrossRefGoogle Scholar
  2. 2.
    Campbell, N.W., Mackeown, W.P.J., Thomas, B.T., Troscianko, T.: Interpreting image databases by region classification. Pattern Recognition (Special Edition on Image Databases) 30 (1997) 555–563CrossRefGoogle Scholar
  3. 3.
    Belongie, S., Carson, C., Greenspan, H., Malik, J.: Color-and texture-based image segmentation using EM and its application to content-based image retrieval. In: Proceedings of IEEE Conference on Computer Vision (ICCV98). (1998) 675–682Google Scholar
  4. 4.
    Zhang, Y.J.: A survey on evaluation mehods for image segmentation. Pattern Recognition 29 (1996) 1335–1346CrossRefGoogle Scholar
  5. 5.
    Levine, M.D., Nazif, A.: Dynamic measurement of computer generated image segmentations. IEEE Transactions on Pattern Analysis and Machine Intelligence 7 (1985) 155–164CrossRefGoogle Scholar
  6. 6.
    Yasnoff, W.A., Mui, J.K., Bacus, J.W.: Error measures for scene segmentation. Pattern Recognition 9 (1977) 217–231CrossRefGoogle Scholar
  7. 7.
    Zhang, Y.J., Gerbrands, J.J.: Objective and quantitative segmentation evaluation and comparison. Signal Processing 39 (1994) 43–54zbMATHCrossRefGoogle Scholar
  8. 8.
    Pareto, V.: Manual of Political Economy. A. M. Kelley, New York (1971) Original French 1906.Google Scholar
  9. 9.
    Barber, C.B., Dobkin, D.P., Huhdanpaa, H.: The Quickhull algorithm for convex hulls. ACM Transactions on Mathematical Software 22 (1996) 469–483MathSciNetzbMATHCrossRefGoogle Scholar
  10. 10.
    Provost, F., Fawcett, T.: Analysis and visualization of classifier performance: Comparison under imprecise class and cost distributions. In Heckerman, D., Mannila, H., Pregibon, D., Uthurusamy, R., eds.: Proceedings of International Conference on Knowledge Discovery and Data Mining (KDD-97), AAAI Press (1997)Google Scholar
  11. 11.
    Everingham, M.R., Muller, H., Thomas, B.T.: Evaluating image segmentation algorithms using monotonic hulls in fitness/cost space. In: Proceedings of British Machine Vision Conference (BMVC2001). (2001) 363–372Google Scholar
  12. 12.
    Corne, D.W., Knowles, J.D.: The Pareto-envelope based selection algorithm for multiobjective optimization. In: Proceedings of International Conference on Parallel Problem Solving from Nature (PPSN VI). (2000) 839–848Google Scholar
  13. 13.
    Silverman, B.W.: Density Estimation for Statistics and Data Analysis. Chapman and Hall (1986)Google Scholar
  14. 14.
    Mackeown, W.P.J.: A Labelled Image Database and its Application to OutdoorSceneAnalysis. PhD thesis, University of Bristol (1994)Google Scholar
  15. 15.
    Collins, D., Wright, W.A., Greenway, P.: The Sowerby image database. In: Proceedings of IEE Conference on Image Processing and its Applications (IPA99). (1999) 306–309Google Scholar
  16. 16.
    Bishop, C.M.: Neural Networks for Pattern Recognition. Clarendon Press, Oxford (1995)Google Scholar
  17. 17.
    Felzenszwalb, P., Huttenlocher, D.: Image segmentation using local variation. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR98). (1998) 98–104Google Scholar
  18. 18.
    Deng, Y., Manjunath, B.S., Shin, H.: Color image segmentation. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR99). (1999) 446–451Google Scholar
  19. 19.
    Ma, W.Y., Manjunath, B.S.: EdgeFlow: A technique for boundary detection and segmentation. IEEE Transactions on Image Processing 9 (2000) 1375–1388MathSciNetzbMATHCrossRefGoogle Scholar
  20. 20.
    Schiele, B., Pentland, A.: Probabilistic object recognition and localization. In: Proceedings of IEEE Conference on Computer Vision (ICCV99). (1999) 177–182Google Scholar
  21. 21.
    Papageorgiou, C., Poggio, T.: A pattern classification approach to dynamical object detection. In: Proceedings of IEEE Conference on Computer Vision (ICCV99). (1999) 1223–1228Google Scholar
  22. 22.
    Yang, M.H., Roth, D., Ahuja, N.: A snow-based face detector. In: Advances in Neural Information Processing Systems 12. MIT Press (2000) 855–861Google Scholar
  23. 23.
    Weber, M., Welling, M., Perona, P.: Unsupervised learning of models for visual object class recognition. In: Proceedings of European Conference on Computer Vision (ECCV00). (2000)Google Scholar
  24. 24.
    Everingham, M.: Methods and Metrics for Image Labelling with Application to Low Vision. PhD thesis, Department of Computer Science, University of Bristol (2002)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2002

Authors and Affiliations

  • Mark Everingham
    • 1
  • Henk Muller
    • 1
  • Barry Thomas
    • 1
  1. 1.Department of Computer ScienceUniversity of BristolBristolUK

Personalised recommendations