Using Parallel Corpora for Multilingual (Multi-document) Summarisation Evaluation
- Cite this paper as:
- Turchi M., Steinberger J., Kabadjov M., Steinberger R. (2010) Using Parallel Corpora for Multilingual (Multi-document) Summarisation Evaluation. In: Agosti M., Ferro N., Peters C., de Rijke M., Smeaton A. (eds) Multilingual and Multimodal Information Access Evaluation. CLEF 2010. Lecture Notes in Computer Science, vol 6360. Springer, Berlin, Heidelberg
We are presenting a method for the evaluation of multilingual multi-document summarisation that allows saving precious annotation time and that makes the evaluation results across languages directly comparable. The approach is based on the manual selection of the most important sentences in a cluster of documents from a sentence-aligned parallel corpus, and by projecting the sentence selection to various target languages. We also present two ways of exploiting inter-annotator agreement levels, apply them both to a baseline sentence extraction summariser in seven languages, and discuss the result differences between the two evaluation versions, as well as a preliminary analysis between languages. The same method can in principle be used to evaluate single-document summarisers or information extraction tools.
Unable to display preview. Download preview PDF.