Skip to main content
Book cover

Language, Culture, Computation. Computing - Theory and Technology pp 409–424Cite as

Benchmarking Applied Semantic Inference: The PASCAL Recognising Textual Entailment Challenges

  • Chapter

Part of the Lecture Notes in Computer Science book series (LNISA,volume 8001)

Abstract

Identifying that the same meaning is expressed by, or can be inferred from, various language expressions is a major challenge for natural language understanding applications such as information extraction, question answering and automatic summarization. Dagan and Glickman [5] proposed Textual Entailment, the task of deciding whether a target text follows from a source text, as a unifying framework for modeling language variability, which has often been addressed in an application-specific manner. In this paper we describe the series of benchmarks developed for the textual entailment recognition task, known as the PASCAL RTE Challenges. As a concrete example, we describe in detail the second RTE challenge, in which our methodology was consolidated, and served as a basis for the subsequent RTE challenges. The impressive success of these challenges established textual entailment as an active research area in natural language processing, attracting a growing community of researchers.

Keywords

  • Question Answering
  • Computational Linguistics
  • Information Extraction System
  • Textual Entailment
  • Entailment System

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-642-45321-2_19
  • Chapter length: 16 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   79.99
Price excludes VAT (USA)
  • ISBN: 978-3-642-45321-2
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   99.99
Price excludes VAT (USA)
Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Baker, C., Fillmore, C., Lowe, J.: The Berkeley Framenet project. In: Proceedings of the COLING-ACL, Montreal, Canada (1998)

    Google Scholar 

  2. Bar-Haim, R., Dagan, I., Dolan, B., Ferro, L., Giampiccolo, D., Magnini, B., Szpektor, I.: The Second PASCAL Recognising Textual Entailment Challenge. In: The Second PASCAL Challenges Workshop on Recognizing Textual Entailment (2006)

    Google Scholar 

  3. Bar-Haim, R., Szpecktor, I., Glickman, O.: Definition and analysis of intermediate entailment levels. In: Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment, Ann Arbor, Michigan, pp. 55–60. Association for Computational Linguistics (June 2005)

    Google Scholar 

  4. Burger, J., Ferro, L.: Generating an entailment corpus from news headlines. In: Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment, Ann Arbor, Michigan, pp. 49–54. Association for Computational Linguistics (June 2005)

    Google Scholar 

  5. Dagan, I., Glickman, O.: Probabilistic textual entailment: Generic applied modeling of language variability. In: PASCAL Workshop on Text Understanding and Mining (2004)

    Google Scholar 

  6. Dagan, I., Glickman, O., Magnini, B.: The PASCAL Recognising Textual Entailment Challenge. In: Quiñonero-Candela, J., Dagan, I., Magnini, B., d’Alché-Buc, F. (eds.) MLCW 2005. LNCS (LNAI), vol. 3944, pp. 177–190. Springer, Heidelberg (2006)

    CrossRef  Google Scholar 

  7. de Marneffe, M.C., MacCartney, B., Grenager, T., Cer, D., Rafferty, A., Manning, C.D.: Learning to distinguish valid textual entailments. In: The Second PASCAL Challenges Workshop on Recognizing Textual Entailment (2006)

    Google Scholar 

  8. Fellbaum, C. (ed.): WordNet: An Electronic Lexical Database. Language, Speech and Communication. MIT Press (1998)

    Google Scholar 

  9. Giampiccolo, D., Magnini, B., Dagan, I., Dolan, B.: The Third PASCAL Recognizing Textual Entailment Challenge. In: Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing (2007)

    Google Scholar 

  10. Giampiccolo, D., Trang Dang, H., Magnini, B., Dagan, I., Dolan, B.: The Fourth PASCAL Recognizing Textual Entailment Challenge. In: Proceedings of the TAC 2008 Workshop (2008)

    Google Scholar 

  11. Glickman, O., Dagan, I., Koppel, M.: A lexical alignment model for probabilistic textual entailment. In: Quiñonero-Candela, J., Dagan, I., Magnini, B., d’Alché-Buc, F. (eds.) MLCW 2005. LNCS (LNAI), vol. 3944, pp. 287–298. Springer, Heidelberg (2006)

    CrossRef  Google Scholar 

  12. Harabagiu, S., Hickl, A., Lacatusu, F.: Satisfying information needs with multi-document summaries. Inf. Process. Manage. 43(6), 1619–1642 (2007)

    CrossRef  Google Scholar 

  13. Hickl, A., Harabagiu, S.: Methods for using textual entailment in open-domain question answering. In: Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL. Association for Computational Linguistics (2006)

    Google Scholar 

  14. Iftene, A., Balahur, A.: Answer validation on English and Romanian languages. In: Peters, C., Deselaers, T., Ferro, N., Gonzalo, J., Jones, G.J.F., Kurimo, M., Mandl, T., Peñas, A., Petras, V. (eds.) CLEF 2008. LNCS, vol. 5706, pp. 448–451. Springer, Heidelberg (2009)

    CrossRef  Google Scholar 

  15. Inkpen, D., Kipp, D., Nastase, V.: Machine learning experiments for textual entailment. In: The Second PASCAL Challenges Workshop on Recognizing Textual Entailment (2006)

    Google Scholar 

  16. Katrenko, S., Adriaans, P.: Using maximal embedded syntactic subtrees for textual entailment recognition. In: The Second PASCAL Challenges Workshop on Recognizing Textual Entailment (2006)

    Google Scholar 

  17. Landis, J.R., Koch, G.G.: The measurements of observer agreement for categorical data. Biometrics 33, 159–174 (1997)

    CrossRef  MATH  Google Scholar 

  18. Nicholson, J., Stokes, N., Baldwin, T.: Detecting entailment using an extended implementation of the basic elements overlap metric. In: The Second PASCAL Challenges Workshop on Recognizing Textual Entailment (2006)

    Google Scholar 

  19. Nielsen, R.D., Ward, W., Martin, J.H.: Classification errors in a domain-independent assessment system. In: Proceedings of the Third Workshop on Innovative Use of Natural Language Processing for Building Educational Applications, at the Forty-Sixth annual meeting of the Association for Computational Linguistics. ACL (2008)

    Google Scholar 

  20. Schilder, F., McInnes, B.T.: Word and tree-based similarities for textual entailment. In: The Second PASCAL Challenges Workshop on Recognizing Textual Entailment (2006)

    Google Scholar 

  21. Vanderwende, L., Dolan, W.B.: What syntax can contribute in the entailment task. In: Quiñonero-Candela, J., Dagan, I., Magnini, B., d’Alché-Buc, F. (eds.) MLCW 2005. LNCS (LNAI), vol. 3944, pp. 205–216. Springer, Heidelberg (2006)

    CrossRef  Google Scholar 

  22. Voorhees, E.M., Harman, D.: Overview of the seventh text retrieval conference. In: Proceedings of the Seventh Text REtrieval Conference (TREC-7). NIST Special Publication (1999)

    Google Scholar 

  23. Zanzotto, F.M., Moschitti, A., Pennacchiotti, M., Pazienza, M.T.: Learning textual entailment from examples. In: The Second PASCAL Challenges Workshop on Recognizing Textual Entailment (2006)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Science Department, Bar-Ilan University, Ramat-Gan, 52900, Israel

    Roy Bar-Haim, Ido Dagan & Idan Szpektor

Authors
  1. Roy Bar-Haim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ido Dagan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Idan Szpektor
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. School of Computer Science, Tel Aviv University, Tel Aviv, Israel

    Nachum Dershowitz

  2. Department of Computing, University of London, Goldsmiths College, 25–27 St. James, New Cross, SE14 6NW, London, UK

    Ephraim Nissan

Rights and permissions

Reprints and Permissions

Copyright information

© 2014 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Bar-Haim, R., Dagan, I., Szpektor, I. (2014). Benchmarking Applied Semantic Inference: The PASCAL Recognising Textual Entailment Challenges. In: Dershowitz, N., Nissan, E. (eds) Language, Culture, Computation. Computing - Theory and Technology. Lecture Notes in Computer Science, vol 8001. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-45321-2_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-45321-2_19

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-45320-5

  • Online ISBN: 978-3-642-45321-2

  • eBook Packages: Computer ScienceComputer Science (R0)