SPARTE, a Test Suite for Recognising Textual Entailment in Spanish

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3878)


The aim of Recognising Textual Entailment (RTE) is to determine whether the meaning of a text entails the meaning of another text named hypothesis. RTE systems can be applied to validate the answers of Question Answering (QA) systems. Once the answer to a question is given by the QA system, a hypothesis is built turning the question plus the answer into an affirmative form. If the text (a given document) entails this hypothesis, then the answer is expected to be correct. Thus, a RTE system becomes an Answer Validation system. Within this framework the first problem is to find collections for training and testing RTE systems. We present here the SPARTE corpus aimed at evaluating RTE systems in Spanish. The paper presents the methodology to build SPARTE from the Spanish QA assessments performed at the Cross-Language Evaluation Forum (CLEF) during the last three editions. The paper also describes the test suite and discusses the appropriate evaluation measures together with their baselines.


Test Suite News Article Question Answering Answer Validation Question Answering System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Barzilay, R., Lee, L.: Learning to Paraphrase: An Unsupervised Approach Using Multiple-Sequence Alignment. In: HLT-NAACL 2003: Main Proceedings, pp. 16–23 (2003)Google Scholar
  2. 2.
    Burger, J., Ferro, L.: Generating an Entailment Corpus from News Headlines. In: Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment, June 2005, pp. 49–54 (2005)Google Scholar
  3. 3.
    Dagan, I., Glickman, O., Magnini, B.: The PASCAL Recognising Textual Entailment Challenge. In: Proceedings of the PASCAL Challenges Workshop on Recognising Textual Entailment, Southampton, UK, April 2005, pp. 1–8 (2005)Google Scholar
  4. 4.
    Dolan, B., Quirk, C., Brockett, C.: Unsupervised Construction of Large Paraphrase Corpora: Exploiting Massively Parallel News Sources. In: Proceedings of COLING 2004, Geneva, Switzerland (2004)Google Scholar
  5. 5.
    Herrera, J., Peñas, A., Verdejo, M.F.: Question Answering Pilot Task at CLEF 2004. In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 581–590. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  6. 6.
    Lin, D., Pantel, P.: DIRT Discovery of inference rules from text. In: KDD 2001: Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 323–328. ACM Press, New York (2001)CrossRefGoogle Scholar
  7. 7.
    Magnini, B., Romagnoli, S., Vallin, A., Herrera, J., Peñas, A., Peinado, V., Verdejo, F., de Rijke, M.: The Multiple Language Question Answering Track at CLEF 2003. In: Peters, C., Gonzalo, J., Braschler, M., Kluck, M. (eds.) CLEF 2003. LNCS, vol. 3237, pp. 471–486. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  8. 8.
    Magnini, B., Vallin, A., Ayache, C., Erbach, G., Peñas, A., de Rijke, M., Rocha, P., Simov, K., Sutcliffe, R.: Overview of the CLEF 2004 Multilingual Question Answering Track. In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 371–391. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  9. 9.
    Peñas, A., Verdejo, M.F., Herrera, J.: Spanish Question Answering Evaluation. In: Gelbukh, A. (ed.) CICLing 2004. LNCS, vol. 2945, pp. 472–483. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  10. 10.
    Shinyama, Y., Sekine, S., Sudo, K., Grishman, R.: Automatic Paraphrase Acquisition from News Articles. In: Automatic Paraphrase Acquisition from News Articles. Proceedings of Human Language Technology Conference, San Diego, USA (2002)Google Scholar
  11. 11.
    Vallin, A., Magnini, B., Giampiccolo, D., Aunimo, L., Ayache, C., Osenova, P., Peñas, A., de Rijke, M., Sacaleanu, B., Santos, D., Sutcliffe, R.: Overview of the CLEF 2005 Multilingual Question Answering Track. In: Proceedings of CLEF 2005 (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  1. 1.Dpto. Lenguajes y Sistemas InformáticosUNED 

Personalised recommendations