Skip to main content
SpringerLink
Log in

A general framework for the evaluation of symbol recognition methods

  • Original Paper
  • Published:
International Journal of Document Analysis and Recognition (IJDAR) Aims and scope Submit manuscript

Abstract

Performance evaluation is receiving increasing interest in graphics recognition. In this paper, we discuss some questions regarding the definition of a general framework for evaluation of symbol recognition methods. The discussion is centered on three key elements in performance evaluation: test data, evaluation metrics and protocols of evaluation. As a result of this discussion we state some general principles to be taken into account for the definition of such a framework. Finally, we describe the application of this framework to the organization of the first contest on symbol recognition in GREC’03, along with the results obtained by the participants.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Aksoy, S., Ye, M., Schauf, M., Song, M., Wang, Y., Haralick, R., Parker, J., Pivovarov, J., Royko, D., Sun, C., Algorithm performance contest. In: Proceedings of 15th International Conference on Pattern Recognition, vol. 4, pp. 870–876, Barcelona, Spain (2000)

  2. Antonacopoulos, A., Gatos, B., Karatzas, D.: ICDAR 2003 page segmentation competition. In: Proceedings of 7th International Conference on Document Analysis and Recognition, Edinburgh (Scotland, UK), pp. 688–689 (2003)

  3. Baird, H.S.: The state of the art of document image degradation modeling. In: Proceedings of 4th IAPR International Workshop on Document Analysis Systems, Rio de Janeiro (Brazil) (2000)

  4. Chhabra A., Phillips I.T. (1998) The 2nd international graphics recognition contest—raster to vector conversion: a report. In: Tombre K., Chhabra A.K. (eds) Graphics Recognition—Algorithms and Systems. Lecture Notes in Computer Science, vol. 1389. Springer, Berlin Heidelberg New York, pp. 390–410

    Google Scholar 

  5. Chhabra A.K. (1998) Graphic symbol recognition: an overview. In: Tombre K., Chhabra A.K. (eds) Graphics Recognition—Algorithms and Systems. Lecture Notes in Computer Science, vol. 1389. Springer, Berlin Heidelberg New York, pp. 68–79

    Google Scholar 

  6. Clark A.F., Courtney P. (2000) Databases for performance characterization. In: Stiehl H.H., Viergever M.A., Vincken K.L. (eds) Performance Characterization in Computer Vision. Kluwer, Dordrecht

    Google Scholar 

  7. Cootes T.F., Taylor C.J., Cooper D.H., Graham J. (1995) Active shape models: Their training and application. Comput. Vis. Image Underst. 61(1): 38–59

    Article  Google Scholar 

  8. Cordella L.P., Vento M. (2000) Symbol recognition in documents: a collection of techniques? Int. J. Doc. Anal. Recognit. 3(2): 73–88

    Article  Google Scholar 

  9. Courtney, P., Thacker, N.A.: Performance characterization in computer vision: the role of statistics in testing and design. In: Blanc-Talon, J., Popescu, D.C. (eds.) Imaging and Vision Systems: Theory, Assessment and Applications. NOVA Science, Hungtington, NY (2003)

  10. Delalandre M., Trupin E., Ogier J., Labiche J. (2005) Contextual system of symbol structural recognition based on an object-process methodology. Electron. Lett. Comput. Vis. Image Anal. 5(2): 16–29

    Google Scholar 

  11. Ghosh D., Shivaprasad A.P. (1999) An analytic approach for generation of artificial hand-printed character database from given generative models. Pattern Recognit. 32, 907–920

    Article  Google Scholar 

  12. Guyon I., Haralick R.M., Hull J.J., Phipliops I.T. (1997) Data sets for OCR and document image understanding research. In: Bunke H., Wang P.S.P. (eds) Handbook of Character Recognition and Document Image Analysis. World Scientific, Singapore, pp. 779–800

    Google Scholar 

  13. Haralick R. (1992) Performance characterization in image analysis: thinning, a case in point. Pattern Recognit. Lett. 13, 5–12

    Article  Google Scholar 

  14. Hilaire, X.: A matching scheme to enhance performance evaluation of raster-to-vector conversion algorithms. In: Proceedings of 7th International Conference on Document Analysis and Recognition, vol. 1, pp. 629–633. Edinburgh, Scotland (2003)

  15. Kanungo, T., Haralick, R.M., Baird, H.S., Stuetzle, W., Madigan, D.: Document degradation models: parameter estimation and model validation. In: Proceedings of IAPR Workshop on Machine Vision Applications, Kawasaki (Japan), pp. 552–557 (1994)

  16. Kanungo T., Haralick R.M., Baird H.S., Stuezle W., Madigan D. (2000) A statistical, nonparametric methodology for document degradation model validation. IEEE Trans. Pattern Anal. Mach. Intell. 22(11): 1209–1223

    Article  Google Scholar 

  17. Lladós J., Valveny E., Sánchez G., Martí E. (2002) Symbol recognition: current advances and perspectives. In: Kwon Y.-B. (eds) Graphics Recognition—Algorithms and Applications. Lecture Notes in Computer Science, vol. 2390. Springer, Berlin Heidelberg New York, pp. 104–127

    Google Scholar 

  18. Lopresti D., Nagy G. (2002) Issues in ground-truthing graphic documents. In: Blostein D., Kwon Y.-B. (eds) Graphics Recognition—Algorithms and Applications. Lecture Notes in Computer Science, vol. 2390. Springer, Berlin Heidelberg New York, pp. 46–66

    Google Scholar 

  19. Lucas S.M., Panaretos A., Sosa L., Tang A., Wong S., Young R., Ashida K., Nagai H., Okamoto M., Yamamoto H., Miyao H., Zhu J., Ou W., Wolf C., Jolion J.M., Todoran L., Worring M., Lin X. (2005) ICDAR 2003 robust reading competitions: entries, results, and future directions. Int. J. Doc. Anal. Recognit. 7(2-3): 105–122

    Article  Google Scholar 

  20. Mariano, V.Y., Min, J., Park, J.-H., Kasturi, R., Mihalcik, D., Li, H., Doermann, D., Drayer, T.: Performance evaluation of object detection algorithms. In: Proceedings of the 16th International Conference on Pattern Recognition, Quebec (Canada), vol. 3, pp. 965–969 (2002)

  21. Philips P.J., Moon H., Rizvi S.A., Rauss P.J. (2000) The evaluation methodology for face-recognition algorithms. IEEE Trans. Pattern Anal. Mach. Intell. 22(10): 1090–1104

    Article  Google Scholar 

  22. Phillips I.T., Chhabra A.K. (1999) Empirical performance evaluation of graphics recognition systems. IEEE Trans. Pattern Anal. Mach. Intell. 21(9): 849–870

    Article  Google Scholar 

  23. Tombre K., Chhabra A.K. (eds) (1998) Graphics Recognition—Algorithms and Systems. Lecture Notes in Computer Science, vol. 1389. Springer, Berlin Heidelberg New York

    Google Scholar 

  24. Valveny E., Dosch Ph. (2004) Performance evaluation of symbol recognition. In: Marinai S., Dengel A. (eds) Document Analysis Systems VI – Proceedings of 6th IAPR International Workshop on Document Analysis Systems, Florence (Italy). Lecture Notes in Computer Science, vol. 3163. Springer, Berlin Heidelberg New York, pp. 354–365

    Google Scholar 

  25. Valveny, E., Dosch, Ph.: Symbol recognition contest: a synthesis. In: Selected Papers from 5th International Workshop on Graphics Recognition, GREC’03. Lecture Notes in Computer Science, vol. 3088, pp. 368–385. Springer, Berlin Heidelberg New York (2004)

  26. Wenyin L., Dori D. (1997) A protocol for performance evaluation of line detection algorithms. Mach. Vis. Appl. 9, 240–250

    Article  Google Scholar 

  27. Wenyin L., Zhai J., Dori D. (2002) Extended summary of the arc segmentation contest. In: Blostein D., Kwon Y.B. (eds) Graphics Recognition: Algorithms and Applications, Selected Papers from 4th International Workshop on Graphics Recognition, GREC’01. Lecture Notes in Computer Science, vol. 2390. Springer, Berlin Heidelberg New York, pp. 343–349

    Google Scholar 

  28. Wilson, C.L., Geist, J., Garris, M.D., Chellappa, R.: Design, integration and evaluation of form-based handprint and OCR systems. Technical report, National Institute of Standards and Technology, Technical Report NISTIR 5932 (1996)

  29. Zhang Y.J. (1996) A survey on evaluation methods for image segmentation. Pattern Recognit. 29(8): 1335–1346

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Centre de Visió per Computador, Edifici O, Campus UAB, Bellaterra (Cerdanyola), 08193, Barcelona, Spain

    E. Valveny

  2. LORIA, 615, rue du jardin botanique, B.P. 101, 54602, Villers-lès-Nancy Cedex, France

    P. Dosch

  3. National University of Ireland, Maynooth, County Kildare, Ireland

    Adam Winstanley & Yu Zhou

  4. Department of Computer Science and Engineering, Fudan University, Shanghai, 200433, China

    Su Yang

  5. Department of Computer Science, City University of Hong Kong, Honk Kong, China

    Luo Yan & Liu Wenyin

  6. University of Nottingham, Nottingham, UK

    Dave Elliman

  7. LITIS Laboratory, Rouen University, Rouen, France

    Eric Trupin & Sébastien Adam

  8. L3i Laboratory, La Rochelle University, Rochelle, France

    Mathieu Delalandre & Jean-Marc Ogier

Authors
  1. E. Valveny
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. P. Dosch
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Adam Winstanley
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yu Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Su Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Luo Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Liu Wenyin
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Dave Elliman
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Mathieu Delalandre
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Eric Trupin
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Sébastien Adam
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Jean-Marc Ogier
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to E. Valveny.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Valveny, E., Dosch, P., Winstanley, A. et al. A general framework for the evaluation of symbol recognition methods. IJDAR 9, 59–74 (2007). https://doi.org/10.1007/s10032-006-0033-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10032-006-0033-x

Keywords

Search

Navigation