On Benchmarking of Invoice Analysis Systems

  • Bertin Klein
  • Stefan Agne
  • Andreas Dengel
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3872)

Abstract

An approach is presented to guide the benchmarking of invoice analysis systems, a specific, applied subclass of document analysis systems. The state of the art of benchmarking of document analysis systems is presented, based on the processing levels: Document Page Segmentation, Text Recognition, Document Classification, and Information Extraction. The restriction to invoices enables and requires a more purposeful, i.e. detailed, targetting of the benchmarking procedures (acquisition of ground truth data, system runs, comparison of data, condensation into meaningful numbers). Therefore the processing of invoices is dissected. The involved data structures are elicited and presented. These are provided, being the building blocks of the actual benchmarking of invoice analysis systems.

Keywords

IEEE Computer Society Document Image Ground Truth Data Private Person Payment Data 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. [ADK03]
    Agne, S., Dengel, A., Klein, B.: Evaluating see — a benchmarking systems for document page segmentation. In: Proceedings of the 7th International Conference on Document Analysis and Recognition IC- DAR 2003, Edinburgh, Scotland, United Kingdom, August, 3-6, vol. I, pp. 634–638 (2003)Google Scholar
  2. [ARR00]
    Agne, S., Rogger, M., Rohrschneider, J.: Benchmarking of document page segmentation. In: Lopresti, D.P., Zhou, J. (eds.) Document and Recognition and Retrieval VII, San Jose, California, USA. Proceedings of SPIE, vol. 3967, pp. 165–171 (2000)Google Scholar
  3. [Bai93]
    Baird, H.S.: Document image defect models and their uses. In: Proceedings of the Second International Conference on Document Analysis and Recognition ICDAR, Tsukuba Science City, Japan, October 20–22, pp. 62–67. IEEE Computer Society Press, Los Alamitos (1993)CrossRefGoogle Scholar
  4. [Bai95]
    Baird, H.S.: Document image defect models. In: O’Gorman, L., Kasturi, R. (eds.) Document Image Analysis, pp. 315–325. IEEE Computer Society Press, Los Alamitos (1995)Google Scholar
  5. [Bre02]
    Breuel, T.M.: Representations and metrics for off-line handwriting segmentation. In: 8th International Workshop on Frontiers in Handwriting Recognition (2002)Google Scholar
  6. [Chi92]
    Chinchor, N.: MUC-4 evaluation metrics. In: Proceedings of the Fourth Message Understanding Conference (MUC-4), McLean, Virginia, USA, June 16–18, pp. 22–29. Morgan Kaufmann Publishers, Inc., San Francisco (1992)Google Scholar
  7. [CS93]
    Chinchor, N., Sundheim, B.: MUC-5 evaluation metrics. In: Proceedings of the Fifth Message Understanding Conference (MUC-5), Baltimore, Maryland, USA, August 25–27, pp. 69–78. Morgan Kaufmann Publishers, Inc., San Francisco (1993)Google Scholar
  8. [DNW+03]
    Dengel, A., Nowak, P., Wagner, C., Rehders, K., Klein, B., Schneider, D., Winkler, M.: Studie automatisierte rechnungseingangsbearbeitung marktpotential, marktübersicht und trends. commercial study (September 2003)Google Scholar
  9. [HB93]
    Ho, T.K., Baird, H.S.: Perfect metrics. In: Proceedings of the Second International Conference on Document Analysis and Recognition ICDAR, Tsukuba Science City, Japan, October 20–22, pp. 593–597. IEEE Computer Society Press, Los Alamitos (1993)CrossRefGoogle Scholar
  10. [HB95]
    Ho, T.K., Baird, H.S.: Evaluation of OCR accuracy using synthetic data. In: Proceedings of the Fourth Annual Symposium on Document Analysis and Information Retrieval SDAIR 1995, Las Vegas, Nevada, April 24–26, pp. 413–422 (1995)Google Scholar
  11. [KBC+05]
    Kumar, A., Burgun, A., Ceusters, W., Cimino, J., Davis, J., Elkin, P., Kalet, I., Rector, A., Rice, J., Rogers, J., Schulz, S., Spackman, K., Zaccagini, D., Zweigenbaum, P., Smith, B.: Six questions on the construction of ontologies in biomedicine. In: AMIA, Washington DC, USA (2005)Google Scholar
  12. [KD04a]
    Klein, B., Dengel, A.: Problem-adaptable document analysis and understanding for high-volume applications. International Journal on Document Analysis and Recognition (2004)Google Scholar
  13. [KD04b]
    Klein, B., Dengel, A.: Results of a study on invoice-reading systems in germany. In: IAPR International Workshop on Document Analysis Systems (2004)Google Scholar
  14. [KDF04]
    Klein, B., Dengel, A., Fordan, A.: Reading and Learning — Adaptive Content Recognition. In: smartFIX: An Adaptive System for Document Analysis and Understanding. LNCS, vol. 2956, pp. 166–186. Springer, Heidelberg (2004)Google Scholar
  15. [KGKD01]
    Klein, B., Gökkus, S., Kieninger, T., Dengel, A.: Three approaches to industrial table spotting. In: Int. Conf. On Document Analysis and Recognition, ICDAR 2001 (2001)Google Scholar
  16. [KRN93]
    Kanai, J., Rice, S.V., Nartker, T.A.: A preliminary evaluation of automatic zoning. In: Grover, K.O. (ed.) Annual Research Report, University of Nevada, Las Vegas, pp. 35–45 (1993), Information Science Research InstituteGoogle Scholar
  17. [KRNN95]
    Kanai, J., Rice, S.V., Nartker, T.A., Nagy, G.: Automated evaluation of OCR zoning. IEEE Transactions on Pattern Analysis and Machine Intelligence 17(1), 86–90 (1995)CrossRefGoogle Scholar
  18. [Lew91]
    Lewis, D.D.: Evaluating text categorization. In: Proceedings of the Workshop on Speech and Natural Language, Pacific Grove, California, USA, February 19–22, pp. 312–318 (1991)Google Scholar
  19. [Lew95]
    Lewis, D.D.: Evaluating and optimizing autonomous text classification systems. In: Fox, E.A., Ingwersen, P., Fidel, R. (eds.) Proceedings of the Eighteenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval SIGIR 1995, Seattle, Washington, USA, July 9–13, pp. 246–254. ACM Press, New York (1995)CrossRefGoogle Scholar
  20. [LS91]
    Lehnert, W., Sundheim, B.: A performance evaluation of textanalysis technologies. AI magazine 12(3), 81–94 (Fall 1991)Google Scholar
  21. [MUC95]
    Proceedings of the Sixth Message Understanding Conference (MUC-6), Columbia, Maryland, USA, November 6–8. Morgan Kaufmann Publishers, Inc. Inhaltsverzeichnis, San Francisco (1995)Google Scholar
  22. [PCL+01]
    Peng, L., Chen, M., Liu, C., Ding, X., Zheng, J.: An automatic performance evaluation method for document page segmentation. In: Proceedings of the Sixth International Conference on Document Analysis and Recognition (ICDAR), Seattle, Washington, USA, September 10-13, pp. 134–137. IEEE Computer Society Press, Los Alamitos (2001)CrossRefGoogle Scholar
  23. [RJN95]
    Rice, S.V., Jenkins, F.R., Nartker, T.A.: The fourth annual test of OCR accuracy. In: Bagdanov, A.D. (ed.) Annual Research Report, University of Nevada, Las Vegas, pp. 11–49 (1995), Information Science Research InstituteGoogle Scholar
  24. [RJN96]
    Rice, S.V., Jenkins, F.R., Nartker, T.A.: The fifth annual test of OCR accuracy. Technical Report TR-96-01, Information Science Research Institute, University of Nevada, Las Vegas, USA (April 1996)Google Scholar
  25. [RKN93]
    Rice, S.V., Kanai, J., Nartker, T.A.: An evaluation of OCR accuracy. In: Grover, K.O. (ed.) Annual Research Report, University of Nevada, Las Vegas, pp. 9–33 (1993), Information Science Research InstituteGoogle Scholar
  26. [RKN94]
    Rice, S.V., Kanai, J., Nartker, T.A.: The third annual test of OCR accuracy. In: Grover, K.O. (ed.) Annual Research Report, University of Nevada, Las Vegas, pp. 11–38 (1994), Information Science Research InstituteGoogle Scholar
  27. [RV94a]
    Randriamasy, S., Vincent, L.: Benchmarking page segmentation algorithms. In: Proceedings of the 1994 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Seattle, Washington, USA, June 21–23, pp. 411–416. IEEE Computer Society Press, Los Alamitos (1994)Google Scholar
  28. [RV94b]
    Randriamasy, S., Vincent, L.: A region-based system for the automatic evaluation of page segmentation algorithms. In: Dengel, A., Spitz, A.L. (eds.) Proceedings of the International Association for Pattern Recognition Workshop on Document Analysis Systems DAS 1994, Kaiserslautern, Germany, October 18–20, pp. 29–41 (1994)Google Scholar
  29. [RVW94]
    Randriamasy, S., Vincent, L., Wittner, B.: An automatic benchmarking scheme for page segmentation. In: Proceedings of the IS&T/SPIE 1994 International Symposium on Electronic Imaging Science and Technology, vol. 2181, pp. 217–230 (1994)Google Scholar
  30. [Sum98]
    Summers, K.: Automatic Discovery Of Logical Document Structure. PhD thesis, Cornell University (1998)Google Scholar
  31. [TMD98]
    Thulke, M., Märgner, V., Dengel, A.: A general approach to quality evaluation of document segmentation results. In: Lee, S.-W., Nakano, Y. (eds.) DAS 1998. LNCS, vol. 1655, pp. 79–88. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  32. [vR79]
    van Rijsbergen, C.J.: Information Retrieval, 2nd edn., Butterworths. Evaluation,  ch. 7, pp. 144–183 (1979)Google Scholar
  33. [WK91]
    Weiss, S.M., Kulikowski, C.A.: Computer Systems That Learn. In: How to Estimate the True Performance of a Learning System,  ch. 2, pp. 17–49. Morgan Kaufmann Publishers, Inc., San Francisco (1991)Google Scholar
  34. [YV95]
    Yanikoglu, B.A., Vincent, L.: Ground-truthing and benchmarking document page segmentation. In: Proceedings of the Third International Conference on Document Analysis and Recognition, Montréal, Canada, August 14–16, vol. 2, pp. 601–604. IEEE Computer Society Press, Los Alamitos (1995)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Bertin Klein
    • 1
  • Stefan Agne
    • 1
  • Andreas Dengel
    • 1
  1. 1.DFKI GmbHKaiserslauternGermany

Personalised recommendations