Skip to main content
SpringerLink
Log in

Object count/area graphs for the evaluation of object detection and segmentation algorithms

  • Original Paper
  • Published:
International Journal of Document Analysis and Recognition (IJDAR) Aims and scope Submit manuscript

Abstract

Evaluation of object detection algorithms is a non-trivial task: a detection result is usually evaluated by comparing the bounding box of the detected object with the bounding box of the ground truth object. The commonly used precision and recall measures are computed from the overlap area of these two rectangles. However, these measures have several drawbacks: they don't give intuitive information about the proportion of the correctly detected objects and the number of false alarms, and they cannot be accumulated across multiple images without creating ambiguity in their interpretation. Furthermore, quantitative and qualitative evaluation is often mixed resulting in ambiguous measures.

In this paper we propose a new approach which tackles these problems. The performance of a detection algorithm is illustrated intuitively by performance graphs which present object level precision and recall depending on constraints on detection quality. In order to compare different detection algorithms, a representative single performance value is computed from the graphs. The influence of the test database on the detection performance is illustrated by performance/generality graphs. The evaluation method can be applied to different types of object detection algorithms. It has been tested on different text detection algorithms, among which are the participants of the ICDAR 2003 text detection competition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Aloimonos, Y., Rosenfeld, A.: REPLY: A response to “ignorance, Myopia and Naiveté in computer vision systems” by R.C. Jain and T.O. Binford. CVGIP: Image Understanding 53(1), 120–124 (1991)

  2. Antonacopoulos, A., Brough, A.: Methodology for flexible and efficient analysis of the performance of page segmentation algorithms. In: Proceedings of the International Conference on Document Analysis and Recognition, pp. 451–454 (1999)

  3. Antonacopoulos, A., Gatos, B., Karatzas, D.: ICDAR 2003 Page Segmentation Competition. In Proceedings of the International Conference on Document Analysis and Recognition, pp. 688–692 (2003)

  4. Bowyer, K.W., Jones, J.P.: REPLY: revolutions and experimental computer vision. CVGIP: Image Understanding 53(1), 125–126 (1991)

    Article  Google Scholar 

  5. Doermann, D., Mihalcik, D.: Tools and techniques for video performance evaluation. In: Proceedings of the International Conference on Pattern Recognition, vol. 4, pp. 4167–4170 (2000)

  6. Fukunaga, K., Hayes, R.R.: Effects of sample size in classifier design. IEEE Trans. Pattern Anal. Machine Intell. 11(8), 873–885 (1989)

    Article  Google Scholar 

  7. Hua, X.-S., Wenyin, L., Zhang, H.-J.: An automatic performance evaluation protocol for video text detection algorithms. IEEE Trans. Circuits Syst. Video Technol. 14(4), 498–507 (2004)

    Article  Google Scholar 

  8. Huang, T.S.: REPLY: computer vision needs more experiments and applications. CVGIP: Image Understanding 53(1), 125–126 (1991)

    Article  Google Scholar 

  9. Huijsmans, N., Sebe, N.: Extended performance graphs for cluster retrieval. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 26–31 (2001)

  10. Jain, R.C., Binford, T.O.: Ignorance, Myopia and Naiveté in computer vision systems. CVGIP: Image Understanding 53(1), 112–117 (1991)

    Article  MATH  Google Scholar 

  11. Landais, R., Vinet, L., Jolion, J.-M.: A goal directed methodology for groundtruthing and evaluating a commercial OCR. Pattern Recognition (submitted) (2004)

  12. Liang, J., Phillips, I.T., Haralick, R.M.: Performance evaluation of document layout analysis algorithms on the UW data set. In Document Recognition IV, Proceedings of the SPIE, pp. 149–160 (1997)

  13. Lucas, S.M., Panaretos, A., Sosa, L., Tang, A., Wong, S., Young, R.: ICDAR 2003 robust reading competitions. In: Proceedings of the Seventh International Conference on Document Analysis and Recognition, vol. 2, pp. 682–687 (2003)

  14. Lucas, S.M., Panaretos, A., Sosa, L., Tang, A., Wong, S., Young, R., Ashida, K., Nagai, H., Okamoto, M., Yamamoto, H., Miyao, H., Zhu, J., Ou, W., Wolf, C., Jolion, J.-M., Todoran, L., Worring, M., Lin, X.: ICDAR 2003 robust reading competitions: entries, results and future directions. International Journal on Document Analysis and Recognition - Special Issue on Camera-based Text and Document Recognition 7(2–3), 105–122 (2005)

    Google Scholar 

  15. Mariano, V.Y., Min, J., Park, J.-H., Kasturi, R., Mihalcik, D., Li, H., Doermann, D., Drayer, T.: Performance evaluation of object detection algorithms. In: Proceedings of the International Conference on Pattern Recognition, vol. 3, pp. 965–969 (2002)

  16. Nagy, G.: Candide's practical principles of experimental pattern recognition. IEEE Trans. Pattern Anal. Machine Intell. 5(2), 199–200 (1983)

    Article  MathSciNet  Google Scholar 

  17. Snyder, M.A.: REPLY: a commentary on the paper by Jain and Binford. CVGIP: Image Understanding 53(1), 118–119 (1991)

    Article  MathSciNet  Google Scholar 

  18. Taylor, G.W., Wolf, C.: Reinforcement learning for parameter control of text detection in images and video sequences. In: Proceedings of the International Conference on Information & Communication Technologies (IEEE), 2004. IEEE Section France (2004)

  19. van Rijsbergen, C.J.: Information retrieval, 2nd edition. Butterworths, London (1979)

  20. Wagner, R.A., Fisher, M.J.: The string to string correction problem. J. Assoc. Comp. Mach. 21(1), 168–173 (1974)

    MATH  Google Scholar 

  21. Wenyin, L., Dori, D.: A protocol for performance evalution of line detection algorithms. Machine Vision and Applications: Special Issue on Performance Evaluation 9(5–6), 240–250 (1997)

    Google Scholar 

  22. Wolf, C.: Text Detection in Images taken from Videos Sequences for Semantic Indexing. PhD thesis, INSA de Lyon, 20, rue Albert Einstein, 69621 Villeurbanne Cedex, France (2003)

  23. Wolf, C., Jolion, J.-M.: Extraction and recognition of artificial text in multimedia documents. Pattern Anal. Appl. 6(4), 309–326 (2003)

    MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. LIRIS-INSA de Lyon, Bât. Jules Verne, 20, Avenue Albert Einstein, 69621, Villeurbanne cedex, France

    Christian Wolf & Jean-Michel Jolion

  2. Laboratoire d'informatique en images et systèmes d'information INSA de Lyon, Villeurbanne, France

    Christian Wolf

Authors
  1. Christian Wolf
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jean-Michel Jolion
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christian Wolf.

Additional information

The work presented in this article has been conceived in the framework of two industrial contracts with France Télécom in the framework of the projects ECAV I and ECAV II with respective numbers 001B575 and 0011BA66.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wolf, C., Jolion, JM. Object count/area graphs for the evaluation of object detection and segmentation algorithms. IJDAR 8, 280–296 (2006). https://doi.org/10.1007/s10032-006-0014-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10032-006-0014-0

Keywords

Search

Navigation