A general framework for the evaluation of symbol recognition methods

  • E. Valveny
  • P. Dosch
  • Adam Winstanley
  • Yu Zhou
  • Su Yang
  • Luo Yan
  • Liu Wenyin
  • Dave Elliman
  • Mathieu Delalandre
  • Eric Trupin
  • Sébastien Adam
  • Jean-Marc Ogier
Original Paper

Abstract

Performance evaluation is receiving increasing interest in graphics recognition. In this paper, we discuss some questions regarding the definition of a general framework for evaluation of symbol recognition methods. The discussion is centered on three key elements in performance evaluation: test data, evaluation metrics and protocols of evaluation. As a result of this discussion we state some general principles to be taken into account for the definition of such a framework. Finally, we describe the application of this framework to the organization of the first contest on symbol recognition in GREC’03, along with the results obtained by the participants.

Keywords

Performance evaluation Symbol recognition 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Aksoy, S., Ye, M., Schauf, M., Song, M., Wang, Y., Haralick, R., Parker, J., Pivovarov, J., Royko, D., Sun, C., Algorithm performance contest. In: Proceedings of 15th International Conference on Pattern Recognition, vol. 4, pp. 870–876, Barcelona, Spain (2000)Google Scholar
  2. 2.
    Antonacopoulos, A., Gatos, B., Karatzas, D.: ICDAR 2003 page segmentation competition. In: Proceedings of 7th International Conference on Document Analysis and Recognition, Edinburgh (Scotland, UK), pp. 688–689 (2003)Google Scholar
  3. 3.
    Baird, H.S.: The state of the art of document image degradation modeling. In: Proceedings of 4th IAPR International Workshop on Document Analysis Systems, Rio de Janeiro (Brazil) (2000)Google Scholar
  4. 4.
    Chhabra A., Phillips I.T. (1998) The 2nd international graphics recognition contest—raster to vector conversion: a report. In: Tombre K., Chhabra A.K. (eds) Graphics Recognition—Algorithms and Systems. Lecture Notes in Computer Science, vol. 1389. Springer, Berlin Heidelberg New York, pp. 390–410Google Scholar
  5. 5.
    Chhabra A.K. (1998) Graphic symbol recognition: an overview. In: Tombre K., Chhabra A.K. (eds) Graphics Recognition—Algorithms and Systems. Lecture Notes in Computer Science, vol. 1389. Springer, Berlin Heidelberg New York, pp. 68–79Google Scholar
  6. 6.
    Clark A.F., Courtney P. (2000) Databases for performance characterization. In: Stiehl H.H., Viergever M.A., Vincken K.L. (eds) Performance Characterization in Computer Vision. Kluwer, DordrechtGoogle Scholar
  7. 7.
    Cootes T.F., Taylor C.J., Cooper D.H., Graham J. (1995) Active shape models: Their training and application. Comput. Vis. Image Underst. 61(1): 38–59CrossRefGoogle Scholar
  8. 8.
    Cordella L.P., Vento M. (2000) Symbol recognition in documents: a collection of techniques? Int. J. Doc. Anal. Recognit. 3(2): 73–88CrossRefGoogle Scholar
  9. 9.
    Courtney, P., Thacker, N.A.: Performance characterization in computer vision: the role of statistics in testing and design. In: Blanc-Talon, J., Popescu, D.C. (eds.) Imaging and Vision Systems: Theory, Assessment and Applications. NOVA Science, Hungtington, NY (2003)Google Scholar
  10. 10.
    Delalandre M., Trupin E., Ogier J., Labiche J. (2005) Contextual system of symbol structural recognition based on an object-process methodology. Electron. Lett. Comput. Vis. Image Anal. 5(2): 16–29Google Scholar
  11. 11.
    Ghosh D., Shivaprasad A.P. (1999) An analytic approach for generation of artificial hand-printed character database from given generative models. Pattern Recognit. 32, 907–920CrossRefGoogle Scholar
  12. 12.
    Guyon I., Haralick R.M., Hull J.J., Phipliops I.T. (1997) Data sets for OCR and document image understanding research. In: Bunke H., Wang P.S.P. (eds) Handbook of Character Recognition and Document Image Analysis. World Scientific, Singapore, pp. 779–800Google Scholar
  13. 13.
    Haralick R. (1992) Performance characterization in image analysis: thinning, a case in point. Pattern Recognit. Lett. 13, 5–12CrossRefGoogle Scholar
  14. 14.
    Hilaire, X.: A matching scheme to enhance performance evaluation of raster-to-vector conversion algorithms. In: Proceedings of 7th International Conference on Document Analysis and Recognition, vol. 1, pp. 629–633. Edinburgh, Scotland (2003)Google Scholar
  15. 15.
    Kanungo, T., Haralick, R.M., Baird, H.S., Stuetzle, W., Madigan, D.: Document degradation models: parameter estimation and model validation. In: Proceedings of IAPR Workshop on Machine Vision Applications, Kawasaki (Japan), pp. 552–557 (1994)Google Scholar
  16. 16.
    Kanungo T., Haralick R.M., Baird H.S., Stuezle W., Madigan D. (2000) A statistical, nonparametric methodology for document degradation model validation. IEEE Trans. Pattern Anal. Mach. Intell. 22(11): 1209–1223CrossRefGoogle Scholar
  17. 17.
    Lladós J., Valveny E., Sánchez G., Martí E. (2002) Symbol recognition: current advances and perspectives. In: Kwon Y.-B. (eds) Graphics Recognition—Algorithms and Applications. Lecture Notes in Computer Science, vol. 2390. Springer, Berlin Heidelberg New York, pp. 104–127Google Scholar
  18. 18.
    Lopresti D., Nagy G. (2002) Issues in ground-truthing graphic documents. In: Blostein D., Kwon Y.-B. (eds) Graphics Recognition—Algorithms and Applications. Lecture Notes in Computer Science, vol. 2390. Springer, Berlin Heidelberg New York, pp. 46–66Google Scholar
  19. 19.
    Lucas S.M., Panaretos A., Sosa L., Tang A., Wong S., Young R., Ashida K., Nagai H., Okamoto M., Yamamoto H., Miyao H., Zhu J., Ou W., Wolf C., Jolion J.M., Todoran L., Worring M., Lin X. (2005) ICDAR 2003 robust reading competitions: entries, results, and future directions. Int. J. Doc. Anal. Recognit. 7(2-3): 105–122CrossRefGoogle Scholar
  20. 20.
    Mariano, V.Y., Min, J., Park, J.-H., Kasturi, R., Mihalcik, D., Li, H., Doermann, D., Drayer, T.: Performance evaluation of object detection algorithms. In: Proceedings of the 16th International Conference on Pattern Recognition, Quebec (Canada), vol. 3, pp. 965–969 (2002)Google Scholar
  21. 21.
    Philips P.J., Moon H., Rizvi S.A., Rauss P.J. (2000) The evaluation methodology for face-recognition algorithms. IEEE Trans. Pattern Anal. Mach. Intell. 22(10): 1090–1104CrossRefGoogle Scholar
  22. 22.
    Phillips I.T., Chhabra A.K. (1999) Empirical performance evaluation of graphics recognition systems. IEEE Trans. Pattern Anal. Mach. Intell. 21(9): 849–870CrossRefGoogle Scholar
  23. 23.
    Tombre K., Chhabra A.K. (eds) (1998) Graphics Recognition—Algorithms and Systems. Lecture Notes in Computer Science, vol. 1389. Springer, Berlin Heidelberg New YorkGoogle Scholar
  24. 24.
    Valveny E., Dosch Ph. (2004) Performance evaluation of symbol recognition. In: Marinai S., Dengel A. (eds) Document Analysis Systems VI – Proceedings of 6th IAPR International Workshop on Document Analysis Systems, Florence (Italy). Lecture Notes in Computer Science, vol. 3163. Springer, Berlin Heidelberg New York, pp. 354–365Google Scholar
  25. 25.
    Valveny, E., Dosch, Ph.: Symbol recognition contest: a synthesis. In: Selected Papers from 5th International Workshop on Graphics Recognition, GREC’03. Lecture Notes in Computer Science, vol. 3088, pp. 368–385. Springer, Berlin Heidelberg New York (2004)Google Scholar
  26. 26.
    Wenyin L., Dori D. (1997) A protocol for performance evaluation of line detection algorithms. Mach. Vis. Appl. 9, 240–250CrossRefGoogle Scholar
  27. 27.
    Wenyin L., Zhai J., Dori D. (2002) Extended summary of the arc segmentation contest. In: Blostein D., Kwon Y.B. (eds) Graphics Recognition: Algorithms and Applications, Selected Papers from 4th International Workshop on Graphics Recognition, GREC’01. Lecture Notes in Computer Science, vol. 2390. Springer, Berlin Heidelberg New York, pp. 343–349Google Scholar
  28. 28.
    Wilson, C.L., Geist, J., Garris, M.D., Chellappa, R.: Design, integration and evaluation of form-based handprint and OCR systems. Technical report, National Institute of Standards and Technology, Technical Report NISTIR 5932 (1996)Google Scholar
  29. 29.
    Zhang Y.J. (1996) A survey on evaluation methods for image segmentation. Pattern Recognit. 29(8): 1335–1346CrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2006

Authors and Affiliations

  • E. Valveny
    • 1
  • P. Dosch
    • 2
  • Adam Winstanley
    • 3
  • Yu Zhou
    • 3
  • Su Yang
    • 4
  • Luo Yan
    • 5
  • Liu Wenyin
    • 5
  • Dave Elliman
    • 6
  • Mathieu Delalandre
    • 8
  • Eric Trupin
    • 7
  • Sébastien Adam
    • 7
  • Jean-Marc Ogier
    • 8
  1. 1.Centre de Visió per ComputadorBarcelonaSpain
  2. 2.LORIAVillers-lès-Nancy CedexFrance
  3. 3.National University of IrelandMaynoothIreland
  4. 4.Department of Computer Science and EngineeringFudan UniversityShanghaiChina
  5. 5.Department of Computer ScienceCity University of Hong KongHonk KongChina
  6. 6.University of NottinghamNottinghamUK
  7. 7.LITIS LaboratoryRouen UniversityRouenFrance
  8. 8.L3i LaboratoryLa Rochelle UniversityRochelleFrance

Personalised recommendations