Advertisement

QA4MRE 2011-2013: Overview of Question Answering for Machine Reading Evaluation

  • Anselmo Peñas
  • Eduard Hovy
  • Pamela Forner
  • Álvaro Rodrigo
  • Richard Sutcliffe
  • Roser Morante
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8138)

Abstract

This paper describes the methodology for testing the performance of Machine Reading systems through Question Answering and Reading Comprehension Tests. This was the attempt of the QA4MRE challenge which was run as a Lab at CLEF 2011–2013. The traditional QA task was replaced by a new Machine Reading task, whose intention was to ask questions that required a deep knowledge of individual short texts and in which systems were required to choose one answer, by analysing the corresponding test document in conjunction with background text collections provided by the organization. Four different tasks have been organized during these years: Main Task, Processing Modality and Negation for Machine Reading, Machine Reading of Biomedical Texts about Alzheimer’s disease, and Entrance Exams. This paper describes their motivation, their goals, their methodology for preparing the data sets, their background collections, their metrics used for the evaluation, and the lessons learned along these three years.

Keywords

Test Document Incorrect Answer Entrance Exam Question Answering Question Answering System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Ferrucci, D., Brown, E., Chu-Carroll, J., Fan, J., Gondek, D., Kalyanpur, A.A., Lally, A., Murdock, J.W., Nyberg, E., Prager, J., Schlaefer, N., Welty, C.: Building Watson: An Overview of the DeepQA Project. AI Magazine 31(3) (2010)Google Scholar
  2. 2.
    Morante, R., Daelemans, W.: Annotating Modality and Negation for a Machine Reading Evaluation. CLEF 2011 Labs and Workshop Notebook Papers (2011)Google Scholar
  3. 3.
    Morante, R., Daelemans, W.: Annotating Modality and Negation for a Machine Reading Evaluation. CLEF 2012 Evaluation Labs and Workshop Online Working Notes (2012)Google Scholar
  4. 4.
    Morante, R., Krallinger, M., Valencia, A., Daelemans, W.: Machine Reading of Biomedical Texts about Alzheimer’s Disease. CLEF 2012 Evaluation Labs and Workshop Online Working Notes (2012)Google Scholar
  5. 5.
    Peñas, A., Rodrigo, Á.: A Simple Measure to Assess Non-response. In: Proceedings of 49th Annual Meeting of the Association for Compu-tational Linguistics-Human Language Technologies (ACL-HLT 2011), Portland, Oregon, USA (2011)Google Scholar
  6. 6.
    Peñas, A., Rodrigo, Á., Sama, V., Verdejo, F.: Overview of the Answer Validation Exercise 2006. In: Peters, C., Clough, P., Gey, F.C., Karlgren, J., Magnini, B., Oard, D.W., de Rijke, M., Stempfhuber, M. (eds.) CLEF 2006. LNCS, vol. 4730, pp. 257–264. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  7. 7.
    Peñas, A., Hovy, E., Forner, P., Rodrigo, Á., Sutcliffe, R., Forascu, C., Sporleder, C.: Overview of QA4MRE at CLEF 2011: Question Answering for Machine Reading Evaluation. Working Notes, CLEF 2011 (2011)Google Scholar
  8. 8.
    Peñas, A., Hovy, E., Forner, P., Rodrigo, Á., Sutcliffe, R., Sporleder, C., Forascu, C., Benajiba, Y., Osenova, P.: Overview of QA4MRE at CLEF 2012: Question Answering for Machine Reading Evaluation. Working Notes, CLEF 2012 (2012)Google Scholar
  9. 9.
    Peñas, A., et al.: Overview of resPubliQA 2009: Question answering evaluation over european legislation. In: Peters, C., Di Nunzio, G.M., Kurimo, M., Mandl, T., Mostefa, D., Peñas, A., Roda, G. (eds.) CLEF 2009. LNCS, vol. 6241, pp. 174–196. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  10. 10.
    Sutcliffe, R., Peñas, A., Hovy, E., Forner, P., Rodrigo, Á., Forascu, C., Benajiba, Y., Osenova, P.: Overview of QA4MRE Main Task at CLEF 2013. Working Notes, CLEF 2013 (2013)Google Scholar
  11. 11.
    Wellner, B., Ferro, L., Greiff, W., Hirschman, L.: Reading Comprehension Tests for Computer-based Understanding Evaluation. Natural Language Engineering 12(4), 305–334 (2006)CrossRefGoogle Scholar
  12. 12.
    Morante, R., Krallinger, M., Valencia, A., Daelemans, W.: Machine Reading of Biomedical Texts about Alzheimer’s Disease 2013. Working Notes, CLEF 2013 (2013)Google Scholar
  13. 13.
    Peñas, A., Miyao, Y., Hovy, E., Forner, P., Kando, N.: Overview of QA4MRE 2013 Entrance Exams Task. Working Notes, CLEF 2013 (2013)Google Scholar
  14. 14.
    Dagan, I., Glickman, O., Magnini, B.: The PASCAL Recognising Textual Entailment Challenge. In: Quiñonero-Candela, J., Dagan, I., Magnini, B., d’Alché-Buc, F. (eds.) MLCW 2005. LNCS (LNAI), vol. 3944, pp. 177–190. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  15. 15.
    Peñas, A., Rodrigo, Á., Verdejo, F.: Overview of the Answer Validation Exercise 2007. In: Peters, C., Jijkoun, V., Mandl, T., Müller, H., Oard, D.W., Peñas, A., Petras, V., Santos, D. (eds.) CLEF 2007. LNCS, vol. 5152, pp. 237–248. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  16. 16.
    Rodrigo, Á., Peñas, A., Verdejo, F.: Overview of the Answer Validation Exercise 2008. In: Peters, C., et al. (eds.) CLEF 2008. LNCS, vol. 5706, pp. 296–313. Springer, Heidelberg (2009)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Anselmo Peñas
    • 1
  • Eduard Hovy
    • 2
  • Pamela Forner
    • 3
  • Álvaro Rodrigo
    • 1
  • Richard Sutcliffe
    • 4
  • Roser Morante
    • 5
  1. 1.NLP&IR GroupUNEDSpain
  2. 2.Carnegie Mellon UniversityUSA
  3. 3.CELCTItaly
  4. 4.School of CSEEUniversity of EssexUK
  5. 5.CLiPSUniversity of AntwerpBelgium

Personalised recommendations