Abstract
A Pilot Question Answering Task has been activated in the Cross-Language Evaluation Forum 2004 with a twofold objective. In the first place, the evaluation of Question Answering systems when they have to answer conjunctive lists, disjunctive lists and questions with temporal restrictions. In the second place, the evaluation of systems’ capability to give an accurate self-scoring about the confidence on their answers. In this way, two measures have been designed to be applied on all these different types of questions and to reward systems that give a confidence score with a high correlation with the human assessments. The forty eight runs submitted to the Question Answering Main Track have been taken as a case of study, confirming that some systems are able to give a very accurate score and showing how the measures proposed reward this fact.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Fukumoto, J., Kato, T., Masui, F.: Question Answering Challenge (QAC-1). An Evaluation of Question Answering Task at NTCIR Workshop 3. In: Oyama, K., Ishida, E., Kando, N. (eds.) Proceedings of the Third NTCIR Workshop on Research in Information Retrieval, Automatic Text Summarization and Question Answering. National Institute of Informatics (2003)
Magnini, B., Romagnoli, S., Vallin, A., Herrera, J., Peñas, A., Peinado, V., Verdejo, F., de Rijke, M.: The Multiple Language Question Answering Track at CLEF 2003. In: Peters, C., Gonzalo, J., Braschler, M., Kluck, M. (eds.) CLEF 2003. LNCS, vol. 3237, pp. 479–495. Springer, Heidelberg (2004)
Magnini, B., Vallin, A., Ayache, C., Erbach, G., Peñas, A., de Rijke, M., Rocha, P., Simov, K., Sutcliffe, R.: Overview of the CLEF 2004 Multilingual Question Answering Track. In: Peters, C., Borri, F. (eds.) Results of the CLEF 2004 Croos-Language System Evaluation Campaign. Working Notes for the CLEF 2004 Workshop, Bath, United Kingdom (2004)
Peñas, A., Herrera, J., Verdejo, F.: Spanish Question Answering Evaluation. In: Gelbukh, A. (ed.) CICLing 2004. LNCS, vol. 2945, pp. 472–483. Springer, Heidelberg (2004)
Pustejovsky, J., Belanger, L., Casta, J., Gaizauskas, R., Hanks, P., Ingria, B., Katz, G., Radev, D., Rumshisky, A., Sanfilippo, A., Sauri, R., Sundheim, B., Verhagen, M.: TERQAS Final Report. Technical Report, MITRE (2002), http://www.cs.brandeis.edu/~jamesp/arda/time/readings.html
Saquete, E., Martínez-Barco, P., Muñoz, R., Vicedo, J.L.: Splitting Complex Temporal Questions for Question Answering Systems. In: Proceedings of the 42nd Meeting of the Association for Computational Linguistics (ACL 2004), Main Volume, pp. 566–573 (2004)
Voorhees, E.M.: The TREC-8 Question Answering Track Report. In: Voorhees, E.M., Harman, D.K. (eds.) Proceedings of the Eigthh Text REtrieval Conference (TREC 8), pp. 77–82. NIST Special Publication 500-246 (1999)
Voorhees, E.M.: Overview of the TREC-9 Question Answering Track. In: Voorhees, E.M., Harman, D.K. (eds.) Proceedings of the Ninth Text REtrieval Conference (TREC 9), pp. 71–79. NIST Special Publication 500-249 (2000)
Voorhees, E.M.: Overview of the TREC 2001 Question Answering Track. In: Voorhees, E.M., Harman, D.K. (eds.) Proceedings of the Tenth Text REtrieval Conference (TREC 2001), pp. 42–51. NIST Special Publication 500-250 (2001)
Voorhees, E.M.: Overview of the TREC 2002 Question Answering Track. In: Voorhees, E.M., Buckland, L.P. (eds.) Proceedings of the Eleventh Text REtrieval Conference (TREC 2002). NIST Special Publication 500-251 (2002)
Voorhees, E.M.: Overview of the TREC 2003 Question Answering Track. In: Proceedings of the Twelfth Text REtrieval Conference (TREC 2003), pp. 54–68. NIST Special Publication 500-255 (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Herrera, J., Peñas, A., Verdejo, F. (2005). Question Answering Pilot Task at CLEF 2004. In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds) Multilingual Information Access for Text, Speech and Images. CLEF 2004. Lecture Notes in Computer Science, vol 3491. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11519645_57
Download citation
DOI: https://doi.org/10.1007/11519645_57
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-27420-9
Online ISBN: 978-3-540-32051-7
eBook Packages: Computer ScienceComputer Science (R0)