Abstract
Traditionally computer vision and pattern recognition algorithms are evaluated by measuring differences between final interpretations and ground truth. These black-box evaluations ignore intermediate results, making it difficult to use intermediate results in diagnosing errors and optimization. We propose “opening the box,” representing vision algorithms as sequences of decision points where recognition results are selected from a set of alternatives. For this purpose, we present a domain-specific language for pattern recognition tasks, the Recognition Strategy Language (RSL). At run-time, an RSL interpreter records a complete history of decisions made during recognition, as it applies them to a set of interpretations maintained for the algorithm. Decision histories provide a rich new source of information: recognition errors may be traced back to the specific decisions that caused them, and intermediate interpretations may be recovered and displayed. This additional information also permits new evaluation metrics that include false negatives (correct hypotheses that the algorithm generates and later rejects), such as the percentage of ground truth hypotheses generated (historical recall), and the percentage of generated hypotheses that are correct(historical precision). We illustrate the approach through an analysis of cell detection in two published table recognition algorithms.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Cesarini, F., Marinai, S., Sarti, L., Soga, G.: Trainable Table Location in Document Images. In: Proc. Sixteenth Int’l Conf. Pattern Recognition, vol. 3, pp. 236–240 (2002)
Competitions in Document Image Analysis Organized by ICDAR and GREC (2008), http://www.icdar2007.org/competition.html
Cordy, J.: The TXL Source Transformation Language. Science of Computer Programming 61(3), 190–210 (2006)
Embley, D., Hurst, M., Lopresti, D., Nagy, G.: Table-processing Paradigms: A Research Survey. Int’l J. Document Analysis and Recognition 8, 66–86 (2006)
Handley, J.: Table Analysis for Multi-line Cell Identification. In: Proc. Doc. Rec. and Retrieval VIII (IS&T/SPIE Electronic Imaging), vol. 4307, pp. 34–43 (2001)
Hu, J., Kashi, R., Lopresti, D., Nagy, G., Wilfong, G.: Why Table Ground-truthing is Hard. In: Proc. Sixth Int’l Conf. Document Analysis and Recognition, pp. 129–133 (2001)
Hu, J., Kashi, R., Lopresti, D., Wilfong, G.: Table Structure Recognition and its Evaluation. In: Proc. Document Rec. and Retrieval VIII, SPIE, vol. 4307, pp. 44–55 (2001)
Hu, J., Kashi, R., Lopresti, D., Wilfong, G.: Evaluating the Performance of Table Processing Algorithms. Int’l J. Document Analysis and Recognition 4(3), 140–153 (2002)
Hurst, M.: Towards a Theory of Tables. Int’l J. Document Analysis and Recognition 8, 123–131 (2006)
Kieninger, T., Dengel, A.: Applying the T-RECS Table Recognition System to the Business Letter Domain. In: Proc. Sixth ICDAR, pp. 518–522 (2001)
Liang, J.: Document Structure Analysis and Performance Evaluation. PhD dissertation, Univ. Washington (1999)
Lopresti, D.: Exploiting WWW Resources in Experimental Document Analysis Research. In: Lopresti, D.P., Hu, J., Kashi, R.S. (eds.) DAS 2002. LNCS, vol. 2423, p. 532. Springer, Heidelberg (2002)
Lopresti, D.P., Nagy, G.: A tabular survey of automated table processing. In: Chhabra, A.K., Dori, D. (eds.) GREC 1999. LNCS, vol. 1941, pp. 93–120. Springer, Heidelberg (2000)
Lopresti, D., Wilfong, G.: Evaluating Document Analysis Results via Graph Probing. In: Proc. Sixth Int’l Conf. Document Analysis and Recognition, pp. 116–120 (2001)
Chen, P.S., Haralick, R.: CD-ROM Document Database Standard. In: Proc. Second Int’l Conf. Document Analysis and Recognition, pp. 478–483 (1993)
Phillips, Chhabra, A.: Empirical Performance Evaluation of Graphics Recognition Systems. IEEE Trans. Pattern Analysis and Machine Intelligence 21(9), 849–870 (1999)
Silva, A.E., Jorge, A., Torgo, L.: Design of an End-to-end Method to Extract Information from Tables. Int’l J. Document Analysis and Recognition 8, 144–171 (2006)
Wang, X.: Tabular Abstraction, Editing and Formatting. PhD dissertation, Univ. Waterloo, Canada (1996)
Wenyin, L., Dori, D.: A Protocol for Performance Evaluation of Line Detection Algorithms. Machine Vision and Applications 9(5/6), 240–250 (1997)
Zanibbi, R.: A Language for Specifying and Comparing Table Recognition Strategies. PhD dissertation, School of Computing, Queen’s Univ., Canada (2004)
Zanibbi, R., Blostein, D., Cordy, J.R.: A Survey of Table Recognition: Models, Observations, Transformations, and Inferences. IJDAR 7(1), 1–16 (2004)
Zanibbi, R., Blostein, D., Cordy, J.R.: Historical Recall and Precision: Summarizing Generated Hypotheses. In: Proc. Eighth ICDAR, vol. 2, pp. 202–206 (2005)
Zanibbi, R., Blostein, D., Cordy, J.R.: Decision-Based Specification and Comparison of Table Recognition Algorithms. In: Machine Learning in Document Analysis and Recognition, pp. 71–103. Springer, Heidelberg (2008)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Zanibbi, R., Blostein, D., Cordy, J.R. (2009). White-Box Evaluation of Computer Vision Algorithms through Explicit Decision-Making. In: Fritz, M., Schiele, B., Piater, J.H. (eds) Computer Vision Systems. ICVS 2009. Lecture Notes in Computer Science, vol 5815. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-04667-4_30
Download citation
DOI: https://doi.org/10.1007/978-3-642-04667-4_30
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-04666-7
Online ISBN: 978-3-642-04667-4
eBook Packages: Computer ScienceComputer Science (R0)