Advertisement

Overview of the CLEF 2006 Multilingual Question Answering Track

  • Bernardo Magnini
  • Danilo Giampiccolo
  • Pamela Forner
  • Christelle Ayache
  • Valentin Jijkoun
  • Petya Osenova
  • Anselmo Peñas
  • Paulo Rocha
  • Bogdan Sacaleanu
  • Richard Sutcliffe
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4730)

Abstract

Having being proposed for the fourth time, the QA at CLEF track has confirmed a still raising interest from the research community, recording a constant increase both in the number of participants and submissions. In 2006, two pilot tasks, WiQA and AVE, were proposed beside the main tasks, representing two promising experiments for the future of QA.Also in the main task some significant innovations were introduced, namely list questions and requiring text snippet(s) to support the exact answers. Although this had an impact on the work load of the organizers both to prepare the question sets and especially to evaluate the submitted runs, it had no significant influence on the performance of the systems, which registered a higher Best accuracy than in the previous campaign, both in monolingual and bilingual tasks. In this paper the preparation of the test set and the evaluation process are described, together with a detailed presentation of the results for each of the languages. The pilot tasks WiQA and AVE will be presented in dedicated articles.

Keywords

Correct Answer Target Language Source Language Exact Answer Previous Campaign 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    QA@CLEF 2006 Organizing Committee. Guidelines (2006), http://clef-qa.itc.it/guidelines.html
  2. 2.
  3. 3.
  4. 4.
    Herrera, J., Peñas, A., Verdejo, F.: Question answering pilot task at CLEF 2004. In: Peters, C., Clough, P.D., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 581–590. Springer, Heidelberg (2005)Google Scholar
  5. 5.
    Magnini, B., Vallin, A., Ayache, C., Erbach, G., Peñas, A., de Rijke, M., Rocha, P., Simov, K., Sutcliffe, R.: Overview of the CLEF 2004 Multilingual Question Answering Track. In: Peters, C., Clough, P.D., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 371–391. Springer, Heidelberg (2005)Google Scholar
  6. 6.
    Rocha, P., Santos, D.: Abrindo a porta á participa internacional em avaliação de RI do português. In: Santos, D. (ed.) Avaliação conjunta: um novo paradigma no processamento computacional da língua portuguesa, IST Press, Lisbon (2006)Google Scholar
  7. 7.
    Santos, D., Rocha, P.: The Key to the First CLEF with Portuguese: Topics, Questions and Answers in CHAVE. In: Peters, C., Clough, P.D., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 821–832. Springer, Heidelberg (2005)Google Scholar
  8. 8.
    Spark Jones, K.: Is question answering a rational task? In: Bernardi, R., Moortgat, M. (eds.) Questions and Answers: Theoretical and Applied Perspectives. Second CoLogNETElsNET Symposium, pp. 24–35. Amsterdam (2003)Google Scholar
  9. 9.
    Vallin, A., Magnini, B., Giampiccolo, D., Aunimo, L., Ayache, C., Osenova, P., Peñas, A., de Rijke, M., Sacaleanu, B., Santos, D., Sutcliffe, R.: Overview of the CLEF 2005 Multilingual Question Answering Track. In: Peters, C., Gey, F.C., Gonzalo, J., Müller, H., Jones, G.J.F., Kluck, M., Magnini, B., de Rijke, M., Giampiccolo, D. (eds.) CLEF 2005. LNCS, vol. 4022, pp. 21–23. Springer, Heidelberg (2006)Google Scholar
  10. 10.
    Voorhees, E. M.: Overview of the TREC 2002 Question Answering Track. In: Voorhees, E.M., Buckland, L.P. (eds) Proceedings of the Eleventh Text Retrieval Conference (TREC 2002 NIST Special Publication 500-251, pp. 115–123. Washington DC (2002) Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Bernardo Magnini
    • 1
  • Danilo Giampiccolo
    • 2
  • Pamela Forner
    • 2
  • Christelle Ayache
    • 3
  • Valentin Jijkoun
    • 4
  • Petya Osenova
    • 5
  • Anselmo Peñas
    • 6
  • Paulo Rocha
    • 7
  • Bogdan Sacaleanu
    • 8
  • Richard Sutcliffe
    • 9
  1. 1.FBK-irst, TrentoItaly
  2. 2.CELCT, TrentoItaly
  3. 3.ELDA/ELRA, ParisFrance
  4. 4.Informatics Institute, University of AmsterdamThe Netherlands
  5. 5.BTBBulgaria
  6. 6.Departamento de Lenguajes y Sistemas Informáticos, UNED, MadridSpain
  7. 7.Linguateca, SINTEF ICT, Norway and Portugal 
  8. 8.DFKIGermany
  9. 9.DLTG, University of LimerickIreland

Personalised recommendations