Skip to main content
Book cover

New Contributions in Information Systems and Technologies pp 13–22Cite as

Question Answering Track Evaluation in TREC, CLEF and NTCIR

  • Conference paper

Part of the Advances in Intelligent Systems and Computing book series (AISC,volume 353)

Abstract

Question Answering (QA) Systems are put forward as a real alternative to Information Retrieval systems as they provide the user with a fast and comprehensible answer to his or her information need. It has been 15 years since TREC introduced the first QA track. The principal campaigns in the evaluation of Information Retrieval have been specific tracks focusing on the development and evaluation of this type of system. This study is a brief review of the TREC, CLEF and NTCIR Conferences from the QA perspective. We present a historical overview of 15 years of QA evaluation tracks using the method of systematic review. We have examined identified the different tasks or specific labs created in each QA track, the types of evaluation question used, as well as the evaluation measures used in the different competitions analyzed. Of the conferences, it is CLEF that has applied the greater variety of types of test question (factoid, definition, list, causal, yes/no, amongst others). NTCIR, held on 13 occasions, is the conference which has made use of a greater number of different evaluation measures. Accuracy, precision and recall have been the three most used evaluation measures in the three campaigns.

Keywords

  • Question Answering Systems
  • Evaluation metrics
  • QA Tracks
  • TREC
  • CLEF
  • NTCIR

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-16486-1_2
  • Chapter length: 10 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   369.00
Price excludes VAT (USA)
  • ISBN: 978-3-319-16486-1
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Dwivedi, S.K., Singh, V.: Research and Reviews in Question Answering System. Procedia Technology 10, 417–424 (2013)

    CrossRef  Google Scholar 

  2. Peñas, A., Magnini, B., Forner, P., Sutcliffe, R., Rodrigo, A., Giampiccolo, D.: Question answering at the cross-language evaluation forum 2003—2010. Lang. Resour. Eval. 46(2), 177–217 (2012)

    CrossRef  Google Scholar 

  3. Mollá, D., Vicedo, J.L.: Question Answering in Restricted Domains: An Overview. Comput. Linguist. 33(1), 41–61 (2007)

    CrossRef  Google Scholar 

  4. Diekerma, A.R., Yilmazel, O., Liddy, E.D.: Evaluation of restricted domain Question-Answering systems Center for Natural Language Processing. Paper 3 (2004)

    Google Scholar 

  5. Sing, G.O., Ardil, C., Wong, W., Sahib, S.: Response Quality Evaluation in Heterogeneous Question Answering System: A Black-box Approach. In: Proceedings of World Academy of Science, Lisbon, vol. 9 (2005)

    Google Scholar 

  6. Olvera-Lobo, M.D., Gutiérrez-Artacho, J.: Question-answering systems as efficient sources of terminological information: an evaluation. Health Information & Libraries Journal 27, 268–276 (2010)

    CrossRef  Google Scholar 

  7. Olvera-Lobo, M.D., Gutiérrez-Artacho, J.: Language resources used in multi‐lingual question‐answering systems. Online Information Review 35(4), 543–557 (2011)

    CrossRef  Google Scholar 

  8. Olvera-Lobo, M.D., Gutiérrez-Artacho, J.: Multilingual Question-Answering System in biomedical domain on the Web: an evaluation. In: Forner, P., Gonzalo, J., Kekäläinen, J., Lalmas, M., de Rijke, M. (eds.) CLEF 2011. LNCS, vol. 6941, pp. 83–88. Springer, Heidelberg (2011)

    CrossRef  Google Scholar 

  9. Olvera-Lobo, M.D., Gutiérrez-Artacho, J.: Evaluation of Open- vs. Restricted- Domain Question Answering Systems in the Biomedical Field. Journal of Information Science 37(2), 152–162 (2011)

    CrossRef  Google Scholar 

  10. Olvera-Lobo, M.D., Gutiérrez-Artacho, J.: Evaluación del rendimiento de los sistemas de búsqueda de respuestas de dominio general. Revista española de Documentación Científica 36(2), e9 (2013)

    Google Scholar 

  11. Radev, D.R., Qi, H., Wu, H., Weiguo, F.: Evaluating Web-based Question Answering Systems. In: Proceedings LREC (2002)

    Google Scholar 

  12. Forner, P., Giampiccolo, D., Magnini, B., Peñas, A., Rodrigo, Á., Sutcliffe, R.: Evaluating multilingual question answering systems at CLEF. In: Proc. 7th International Conference on Language Resources and Evaluation. LREC 2010, Malta, pp. 2774–2781 (2010)

    Google Scholar 

  13. Voorhees, E.: Overview of TREC 2003. In: Proceedings of the 12th Text Retrieval Conference (2003)

    Google Scholar 

  14. Kelly, D., Sugimoto, C.R.: A systematic review of interactive information retrieval evaluation studies, 1967–2006. Journal of the American Society for Information Science and Technology 64, 745–770 (2013)

    CrossRef  Google Scholar 

  15. Moghadasi, S.I., Ravana, S.D.: Low-cost evaluation techniques for information retrieval systems: A review. Journal of Informetrics 7(2), 301–312 (2013)

    CrossRef  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Information and Communication, Colegio Máximo de Cartuja, University of Granada, Campus Cartuja s/n, 18071, Granada, Spain

    María-Dolores Olvera-Lobo

  2. CSIC, Unidad Asociada Grupo SCImago, Madrid, Spain

    María-Dolores Olvera-Lobo

  3. Department of Translation and Interpreting, Faculty of Translation and Interpreting, University of Granada, C/ Buensuceso, 11, 18003, Granada, Spain

    Juncal Gutiérrez-Artacho

Authors
  1. María-Dolores Olvera-Lobo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Juncal Gutiérrez-Artacho
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to María-Dolores Olvera-Lobo .

Editor information

Editors and Affiliations

  1. DEI/FCT, Universidade de Coimbra, Coimbra, Portugal

    Alvaro Rocha

  2. Campus de Campolide, ISEGI, Lisboa, Portugal

    Ana Maria Correia

  3. DEIS, Università della Calabria, Arcavacata di Rende, Italy

    Sandra Costanzo

  4. DIS, Universidade do Minho, Guimarães, Portugal

    Luis Paulo Reis

Rights and permissions

Reprints and Permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Olvera-Lobo, MD., Gutiérrez-Artacho, J. (2015). Question Answering Track Evaluation in TREC, CLEF and NTCIR. In: Rocha, A., Correia, A., Costanzo, S., Reis, L. (eds) New Contributions in Information Systems and Technologies. Advances in Intelligent Systems and Computing, vol 353. Springer, Cham. https://doi.org/10.1007/978-3-319-16486-1_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-16486-1_2

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-16485-4

  • Online ISBN: 978-3-319-16486-1

  • eBook Packages: Computer ScienceComputer Science (R0)