Evaluation Protocol for Localization Metrics

Application to a Comparative Study
  • Baptiste Hemery
  • Hélène Laurent
  • Christophe Rosenberger
  • Bruno Emile
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5099)


Localization metrics permit to quantify the correctness of object detection in an image interpretation result. This paper deals with the definition of a protocol in order to evaluate the behavior of localization metrics. We first define some properties that metrics should verify and create a synthetic database that enables to verify those properties on different metrics. After presenting the tested localization metrics, the results obtained following the proposed protocol are exposed. Finally, some conclusions and perspectives are given.


Ground Truth Localization Algorithm Localize Object Localization Result Segmentation Evaluation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Everingham, M., Zisserman, A., Williams, C., Van Gool, L., Allan, M., Bishop, C., Chapelle, O., Dalal, N., Deselaers, T., Dorko, G., et al.: The 2005 pascal visual object classes challenge (2005)Google Scholar
  2. 2.
    D’Angelo, E., Herbin, S., Ratiéville, M.: Robin challenge evaluation principles and metrics (2006)Google Scholar
  3. 3.
    Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: 8th Int’l Conf. Computer Vision, pp. 416–423 (2001)Google Scholar
  4. 4.
    Odet, C., Belaroussi, B., Benoit-Cattin, H. (eds.): Scalable Discrepancy Measures for Segmentation Evaluation, vol. 1 (2002)Google Scholar
  5. 5.
    Pratt, W., Faugeras, O.D., Gagalowicz, A.: Visual discrimination of stochastic texture fields. IEEE Transactions on Systems, Man, and Cybernetics 8(11), 796–804 (1978)CrossRefGoogle Scholar
  6. 6.
    Yasnoff, W.A., Mui, J.K., Bacus, J.W.: Error measures for scene segmentation. Pattern Recognition 9, 217–231 (1977)CrossRefGoogle Scholar
  7. 7.
    Baddeley, A.J.: An error metric for binary images. Robust Computer Vision, 59–78 (1992)Google Scholar
  8. 8.
    Chabrier, S.: Contribution à l’évaluation de performances en segmentation d’images. PhD thesis, Université d’Orléans (2005)Google Scholar
  9. 9.
    Wilson, D.L., Baddeley, A.J., Owens, R.A.: A new metric for grey-scale image comparison. International Journal of Computer Vision 24(1), 5–17 (1997)CrossRefGoogle Scholar
  10. 10.
    Henricsson, O., Baltsavias, E.: 3-d building reconstruction with aruba: A qualitative and quantitative evaluation. In: Automatic Extraction of Man-Made Objects from Aerial and Space Images (II), pp. 65–76 (1997)Google Scholar
  11. 11.
    Coquin, D., Bolon, P., Chehadeh, Y.: Evaluation quantitative d’images filtrées. In: GRETSI 1997, vol. 2, pp. 1351–1354 (1997)Google Scholar
  12. 12.
    Basseville, M.: Distance measures for signal processing and pattern recognition. Signal Processing 18(4), 349–369 (1989)CrossRefMathSciNetGoogle Scholar
  13. 13.
    Huang, Q., Dom, B.: Quantitative methods of evaluating image segmentation. In: International Conference on Image Processing (ICIP 1995), Washington DC, USA, vol. 3, pp. 53–56 (1995)Google Scholar
  14. 14.
    Peli, T., Malah, D.: A study of edge detection algorithms. Comp. Graphics Image (1982)Google Scholar
  15. 15.
    Hafiane, A.: Caractérisation de textures et segmentation pour la recherche d’images par le contenu. PhD thesis, Université de Paris-Sud XI (2005)Google Scholar
  16. 16.
    Hafiane, A., Chabrier, S., Rosenberger, C., Laurent, H.: A new supervised evaluation criterion for region based segmentation methods. In: Blanc-Talon, J., Philips, W., Popescu, D., Scheunders, P. (eds.) ACIVS 2007. LNCS, vol. 4678, pp. 439–448. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  17. 17.
    Vinet, L.: Segmentation et mise en correspondance de régions de paires d’images stéréoscopiques. PhD thesis, Université de Paris IX Dauphine (1991)Google Scholar
  18. 18.
    Cocquerez, J.P., Philipp, S.: Analyse d’Images: filtrage et segmentation. Masson (1995)Google Scholar
  19. 19.
    Beauchemin, M., Thomson, K.B., Edwards, G.: On the hausdorff distance used for the evaluation of segmentation results. Canadian Journal of Remote Sensing 24(1), 3–8 (1998)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Baptiste Hemery
    • 1
  • Hélène Laurent
    • 1
  • Christophe Rosenberger
    • 2
  • Bruno Emile
    • 1
  1. 1.Institut Prisme, ENSI de BourgesUniversité d’OrléansBourgesFrance
  2. 2.Laboratoire Greyc, ENSICAENUniversité de Caen - CNRSCaenFrance

Personalised recommendations