Abstract
Different metrics have been proposed for the estimation of how good a parser-produced syntactic tree is when judged by a correct tree from the treebank. The emphasis of measurement has been on the number of correct constituents in terms of constituent labels and bracketing accuracy. This article proposes the use of the NIST scheme as a better alternative for the evaluation of parser output in terms of correct match, substitution, deletion, and insertion. It describes an experiment to measure the performance of the Survey Parser that was used to complete the syntactic annotation of the International Corpus of English. This article will finally report empirical scores for the performance of the parser and outline some future research.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Sutcliffe, R.F.E., Koch, H.-D., McElligott, A. (eds.): Industrial Parsing of Software Manuals. Rodopi, Amsterdam (1996)
Carroll, J.A., Briscoe, E.J., Sanfilippo, A.: Parser evaluation: a survey and a new proposal. In: Proceedings of the First International Conference on Language Resources and Evaluation, Granada, Spain, May 28-30, pp. 447–54 (1998)
Gaizauskas, R., Hepple, M., Huyck, C.: A scheme for comparative evaluation of diverse parsing systems. In: Proceedings of the First International Conference on Language Resources and Evaluation, Granada, Spain, May 28-30 (1998)
Lin, D.: A dependency-based method for evaluating broad-coverage parsers. Natural Language Engineering 4, 97–114 (1998)
Srinivasan, B., Sarkar, A., Doran, C., Hockey, B.A.: Grammar and parser evaluation in the XTAG project. In: Carroll, Basili, et al. (eds.) Proceedings of the Workshop on the Evaluation of Parsing Systems, First International Conference on Language Resources and Evaluation, Granada, Spain, May 26(1998)
Pallett, D.S., Fisher, W.M., Fiscus, J.G.: Tools for the Analysis of Benchmark Speech Recognition Tests. In: Proceedings of ICASSP 1990, pp. 97–100 (1990)
Pallet, D.S., Garofolo, J.S., Fiscus, J.G.: Measurements in Support of Research Accomplishments. Communications of the ACM 43(2), 75–79 (2000)
Greenbaum, S.: A new corpus of English: ICE. In: Svartvik, J. (ed.) Directions in Corpus Linguistics: Proceedings of Nobel Symposium, Stockholm, August 4-8, 1991, vol. 82, pp. 171–179. Mouton de Gruyter, Berlin (1992)
Greenbaum, S.: The International Corpus of English. Oxford University Press, Oxford (1996)
Fang, A.C.: The Survey Parser: Design and development. In: Greenbaum, S. (ed.), pp. 142–160 (1996)
Fang, A.C.: From Cases to Rules and Vice Versa: Robust Practical Parsing with Analogy. In: Proceedings of the Sixth International Workshop on Parsing Technologies, Trento, Italy, February 23-25, pp. 77–88 (2000)
Fang, A.C.: The Syntactically Annotated ICE Corpus and the Automatic Induction of a Formal Grammar. In: Proceedings of 6th International Workshop on Linguistically Interpreted Corpora, Jeju Island, Korea, October 15, vol. 15 (2005)
Charniak, E.: A maximum-entropy-inspired parser. In: Proceedings of the 1st Annual Meeting of the North American Chapter of the ACL, Seattle, Washington (2000)
Gildea, D.: Corpus variation and parser performance. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (2001)
Klein, D., Manning, C.: Accurate unlexicalized parsing. In: Proceedings of the 41st Annual Meeting of the ACL, July 2003, pp. 423–430 (2003)
Vilares, M., Ribadas, F.J., Vilares, J.: Phrase Similarity through the Edit Distance. In: Galindo, F., Takizawa, M., Traunmüller, R. (eds.) DEXA 2004. LNCS, vol. 3180, pp. 306–317. Springer, Heidelberg (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Fang, A.C. (2006). Evaluating the Performance of the Survey Parser with the NIST Scheme. In: Gelbukh, A. (eds) Computational Linguistics and Intelligent Text Processing. CICLing 2006. Lecture Notes in Computer Science, vol 3878. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11671299_19
Download citation
DOI: https://doi.org/10.1007/11671299_19
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-32205-4
Online ISBN: 978-3-540-32206-1
eBook Packages: Computer ScienceComputer Science (R0)