Enhancing N-Gram-Based Summary Evaluation Using Information Content and a Taxonomy

  • Mijail Kabadjov
  • Josef Steinberger
  • Ralf Steinberger
  • Massimo Poesio
  • Bruno Pouliquen
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5993)

Abstract

In this paper we propose a novel information-theoretic metric for automatic summary evaluation when model summaries are available as in the setting of the AESOP task of the Update Summarization track of the Text Analysis Conference (TAC). The metric is based on the concept of information content operationalized by using a taxonomy. Hereby, we present and discuss the results obtained at TAC 2009.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Lin, C.-Y., Hovy, E.: Automatic evaluation of summaries using n-gram co-occurrence statistics. In: Proceedings of HLT-NAACL, Edmonton, Canada (2003)Google Scholar
  2. Nenkova, A., Passonneau, R.: Evaluating content selection in summarization: The pyramid method. In: Proceedings of the Meeting of the North American Chapter of the Association for Computational Linguistics, NAACL (2004)Google Scholar
  3. Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: BLEU: a method for automatic evaluation of machine translation. Technical Report RC22176 (W0109-022), IBM Corporation (2001)Google Scholar
  4. Resnik, P.: Using information content to evaluate semantic similarity in a taxonomy. In: Proceedings of the National Conference on Artificial Intelligence, AAAI (November 1995)Google Scholar
  5. Steinberger, J., Kabadjov, M., Pouliquen, B., Steinberger, R., Poesio, M.: WB-JRC-UT’s participation in TAC 2009: Update summarization and AESOP tasks. In: National Institute of Standards and Technology (eds.) Proceedings of the Text Analysis Conference, Gaithersburg, MD (November 2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Mijail Kabadjov
    • 1
  • Josef Steinberger
    • 1
  • Ralf Steinberger
    • 1
  • Massimo Poesio
    • 2
    • 3
  • Bruno Pouliquen
    • 1
  1. 1.Joint Research CentreEuropean CommissionIspraItaly
  2. 2.University of EssexColchesterUnited Kingdom
  3. 3.Universitá di TrentoPovoItaly

Personalised recommendations