Evaluating the Performance of Discourse Parser Systems

Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 356)

Abstract

In this paper are analyzed two different methods for comparing discourse tree structures. The aim is to identify the best technique to evaluate the output of discourse parser system. The first, is a traditional method and is based on quantifying the similarities between the parser output and a reference discourse tree (gold tree). It uses three scores: Precision, Recall, and F-measures. The second one examines the discourse tree representations in qualitative terms. Like first one, three scores are computed: Overlapping score, for analyzing the similarities in terms of topology, Nuclearity scores which take into account the type of nodes, and Veins scores which measure the coherence of discourse. By evaluating discourse tree structure resulted from a parser, the performance of discourse parser systems can be measured. A comparative survey is realized aiming to present assessment methods for discourse tree structures.

Keywords

Discourse parser Evaluating methods Discourse trees 

Notes

Acknowledgments

This paper is supported by the Sectoral Operational Programme Human Resources Developed (SOP HRD), financed from the European Social Fund and by the Romanian Government under the contract number POSDRU/159/1.5/S/133675.

References

  1. 1.
    Langer H (2001) Syntax and parsing. In: Carstensen K-U, Ebert C, Endriss C, Jekat S, Klabunde R, Langer H (eds) Computerlinguistik und Sprachtechnologie. Eine Einführung, pp 203–245. Heidelberg, Berlin: Spektrum Akademischer VerlagGoogle Scholar
  2. 2.
    Soricut R, Marcu D (2003) Sentence level discourse parsing using syntactic and lexical information. In: Proceedings of the HLT/NAACL, pp 228–235Google Scholar
  3. 3.
    Kermes H (2003) On-line text analysis for computational lexicography. Ph.D. thesis, IMS, University of Stuttgart. Arbeitspapiere des Instituts für Maschinelle Sprachverarbeitung (AIMS), vol 9, number 3Google Scholar
  4. 4.
    Eckle-Kohler J (1999) Linguistisches Wissen zur automatischen Lexicon-Akquisition aus deutschen Textcorpora. Ph. D. thesis, Universität Stuttgart: IMSGoogle Scholar
  5. 5.
    Marcu D (2000) The rhetorical parsing of unrestricted texts: a surface-based approach. Comput Linguist 26(3):395–448CrossRefGoogle Scholar
  6. 6.
    Mitocariu E, Anechitei DA, Cristea D (2013) Comparing discourse tree structures. In: Gelbukh A (ed) Computational linguistics and intelligent text processing 1, vol 7816Google Scholar
  7. 7.
    Mann WC, Thompson SA (1998) Rhetorical structure theory: toward a functional theory of text organization. Text 8(3):243–81Google Scholar
  8. 8.
    Cristea D, Ide N, Romary L (1998) Veins theory: a model of global discourse cohesion and coherence. In: Proceedings of the 17th international conference on computational linguisticsGoogle Scholar
  9. 9.
    Black E (1992) Meeting of interest group on evaluation of broad-coverage grammars of English. LINGUIST List 3.587. http://www.linguistlist.org/issues/3/3-587.html
  10. 10.
    Carroll JA, Briscoe E, Sanlippo A (1998) Parser evaluation: a survey and a new proposal. In: Proceedings of the 1st international conference on language resources and evaluation, Granada, Spain, pp 447–454Google Scholar
  11. 11.
    Briscoe EJ, Carroll JA, Copestake A (2002) Relational evaluation schemes. In: Proceedings workshop beyond parseval—towards improved evaluation measures for parsing systems, 3rd international conference on language resources and evaluation, pp 4–38. Las Palmas, Canary IslandsGoogle Scholar
  12. 12.
    Sampson Geoffrey, Babarczy Anna (2003) A test of the leaf-ancestor metric for parse accuracy. Nat Lang Eng 9(4):365–380CrossRefGoogle Scholar
  13. 13.
    Maziero EG, Pardo TAS (2009) Metodologia de avaliação automática de estruturas retóricas [methodology for automatic evaluation of rhetorical structures]. In: 7th Brazilian symposium in information and human language technology (STIL), São Carlos, BrazilGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.Faculty of Computer Science“Al.I.Cuza” University of IasiIasiRomania
  2. 2.SOP HRD/159/1.5/S/133675 ProjectRomanian Academy Iasi Branch (POSDRU/159/1.5/S/133675)IasiRomania

Personalised recommendations