Language Resources and Evaluation

, Volume 46, Issue 2, pp 177–217 | Cite as

Question answering at the cross-language evaluation forum 2003–2010

  • Anselmo Peñas
  • Bernardo Magnini
  • Pamela Forner
  • Richard Sutcliffe
  • Álvaro Rodrigo
  • Danilo Giampiccolo
Original Paper

Abstract

The paper offers an overview of the key issues raised during the 8 years’ activity of the Multilingual Question Answering Track at the Cross Language Evaluation Forum (CLEF). The general aim of the track has been to test both monolingual and cross-language Question Answering (QA) systems that process queries and documents in several European languages, also drawing attention to a number of challenging issues for research in multilingual QA. The paper gives a brief description of how the task has evolved over the years and of the way in which the data sets have been created, presenting also a short summary of the different types of questions developed. The document collections adopted in the competitions are outlined as well, and data about participation is provided. Moreover, the main measures used to evaluate system performances are explained and an overall analysis of the results achieved is presented.

Keywords

Question answering Evaluation CLEF 

References

  1. Agirre, E., Ansa, O., Arregi, X., Lopez de Lacalle, M., Otegi, A., Saralegi, Z., et al. (2009). ElhuyarIXA: Semantic relatedness and crosslingual passage retrieval. In C. Peters, G. di Nunzio, M. Kurimo, Th. Mandl, D. Mostefa, A. Peñas, & G. Roda (Eds.), Multilingual information access evaluation vol. I text retrieval experiments, Workshop of the cross-language evaluation forum, CLEF 2009, Corfu, Greece, 30 September–2 October. Lecture notes in computer science 6241. Springer (Revised selected papers).Google Scholar
  2. Agirre, E., Ansa, O., Arregi, X., Lopez de Lacalle, M., Otegi, A., & Saralegi, X. (2010). Document expansion for cross-lingual passage retrieval. In M. Braschler, D. Harman, & E. Pianta (Eds.), Notebook papers of CLEF 2010 LABs and workshops, September 22–23, 2010 Padua, Italy.Google Scholar
  3. Amaral, C., Figueira, H., Martins, A., Mendes, A., Mendes, P., & Pinto, C. (2005). Priberam’s question answering system for Portuguese. In C. Peters, F. C. Gey, J. Gonzalo, H. Müller, G. J. F. Jones, M. Kluck, B. Magnini, & M. de Rijke (Eds.), Accessing multilingual information repositories, 6th workshop of the cross-language evaluation forum, CLEF 2005, Vienna, Austria, September 21–23, 2005 (Revised selected papers).Google Scholar
  4. Amaral, C., Cassan, A., Figueira, H., Martins, A., Mendes, A., Mendes, P., et al. (2007). Priberam’s question answering system in QA@CLEF 2007. In C. Peters, V. Jijkoun, T. Mandl, H. Müller, D. W. Oard, A. Peñas, & D. Santos (Eds.), Advances in multilingual and multimodal information retrieval, 8th workshop of the cross-language evaluation forum, CLEF 2007, Budapest, Hungary, September 19–21, 2007 (Revised selected papers).Google Scholar
  5. Amaral, C., Cassan, A., Figueira, H., Martins, A., Mendes, A., Mendes, P., et al. (2008). Priberam’s question answering system in QA@CLEF 2008. In C. Peters, T. Mandl, V. Petras, A. Peñas, H. Müller, D. Oard, V. Jijkoun, & D. Santos (Eds.), Evaluating systems for multilingual and multimodal information access, 9th workshop of the cross-language evaluation forum, CLEF 2008, Aarhus, Denmark, September 17–19, 2008 (Revised selected papers).Google Scholar
  6. Bouma, G., Mur, J., van Noord, G., van der Plas, L., & Tiedemann, J. (2005). Question answering for Dutch using dependency relations. In C. Peters, F. C. Gey, J. Gonzalo, H. Müller, G. J. F. Jones, M. Kluck, B. Magnini, & M. de Rijke (Eds.), Accessing multilingual information repositories, 6th workshop of the cross-language evaluation forum, CLEF 2005, Vienna, Austria, September 21–23, 2005 (Revised selected papers).Google Scholar
  7. Bowden, M., Olteanu, M., Suriyentrakorn, P., d’Silva, T., & Moldovan. D. (2007). Multilingual question answering through intermediate translation: LCC’s PowerAnswer at QA@CLEF 2007. In C. Peters, V. Jijkoun, T. Mandl, H. Müller, D.W. Oard, A. Peñas, & D. Santos (Eds.), Advances in multilingual and multimodal information retrieval, 8th workshop of the cross-language evaluation forum, CLEF 2007, Budapest, Hungary, September 19–21, 2007 (Revised selected papers).Google Scholar
  8. Cassan, A., Figueira, H., Martins, A., Mendes, A., Mendes, P., Pinto, C., et al. (2006). Priberam’s question answering system in a cross-language environment. In C. Peters, P. Clough, F. C. Gey, J. Karlgren, B. Magnini, D. W. Oard, M. de Rijke, & M. Stempfhuber (Eds.), Evaluation of multilingual and multi-modal information retrieval, 7th workshop of the cross-language evaluation forum, CLEF 2006, Alicante, Spain, September 20–22, 2006 (Revised selected papers).Google Scholar
  9. Clarke, C., Cormack, G., & Lynam, T. (2001). Exploiting redundancy in question answering. In Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR-2001). September 9–13, 2001, New Orleans, Louisiana. ACM 2001, ISBN 1-58113-331-6.Google Scholar
  10. Correa, S., Buscaldi, D., & Rosso, P. (2009). NLEL-MAAT at CLEF-ResPubliQA. In C. Peters, G. di Nunzio, M. Kurimo, Th. Mandl, D. Mostefa, A. Peñas, & G. Roda (Eds.), Multilingual information access evaluation vol. I text retrieval experiments, workshop of the cross-language evaluation forum, CLEF 2009, Corfu, Greece, September 30–October 2, 2010. Lecture notes in computer science 6241. Springer (Revised selected papers).Google Scholar
  11. Dornescu, I., Puscasu, G., & Orsasan, C. (2008). University of Wolverhampton at CLEF 2008. In C. Peters, T. Mandl, V. Petras, A. Peñas, H. Müller, D. Oard, V. Jijkoun, & D. Santos (Eds.), Evaluating systems for multilingual and multimodal information access, 9th workshop of the cross-language evaluation forum, CLEF 2008, Aarhus, Denmark, September 17–19, 2008 (Revised selected papers).Google Scholar
  12. Ferrandez, O., Spurk, C., Kouylekov, M., Dornescu, I., Ferrandez, S., Negri, M., Izquierdo, R., Tomas, D., Orasan, C., Neumann, G., Magnini B., & Vicedo, J. L. (2011). The QALL-ME framework: A specifiable-domain multilingual question answering architecture. Web semantics: Science, services and agents on the world wide web, 9 (2), Provenance in the Semantic Web, 137–145.Google Scholar
  13. Ferrucci, D., Brown, E., Chu-Carroll, J., Fan, J., Gondek, D., Kalyanpur, A. A., et al. (2010). Building Watson: An overview of the DeepQA project. AI Magazine, 31(3), 59–79.Google Scholar
  14. Forner, P., Peñas, A., Alegria, I., Forascu, C., Moreau, N., Osenova, P., et al. (2008). Overview of the CLEF 2008 multilingual question answering track. In C. Peters, T. Mandl, V. Petras, A. Peñas, H. Müller, D. Oard, V. Jijkoun, & D. Santos (Eds.), Evaluating systems for multilingual and multimodal information access, 9th workshop of the cross-language evaluation forum, CLEF 2008, Aarhus, Denmark, September 17–19, 2008 (Revised selected papers).Google Scholar
  15. Giampiccolo, D., Forner, P., Herrera, J., Peñas, A., Ayache, C., Forascu, C., et al. (2007). Overview of the CLEF 2007 Multilingual Question Answering Track. In C. Peters, V. Jijkoun, T. Mandl, H. Müller, D.W. Oard, A. Peñas, & D. Santos (Eds.), Advances in multilingual and multimodal information retrieval, 8th workshop of the cross-language evaluation forum, CLEF 2007, Budapest, Hungary, September 19–21, 2007 (Revised selected papers).Google Scholar
  16. Gillard, L., Sitbon, L., Blaudez, E., Bellot, P., & El-Bèze, M. (2006). The LIA at QA@CLEF-2006. In C. Peters, P. Clough, F. C. Gey, J. Karlgren, B. Magnini, D. W. Oard, M. de Rijke, & M. Stempfhuber (Eds.), Evaluation of multilingual and multi-modal information retrieval, 7th workshop of the cross-language evaluation forum, CLEF 2006, Alicante, Spain, September 20–22, 2006 (Revised selected papers).Google Scholar
  17. Gonzalo, J., Clough, P., & Vallin, A. (2005). Overview of the CLEF 2005 interactive track. In C. Peters, F. C. Gey, J. Gonzalo, H. Müller, G. J. F. Jones, M. Kluck, B. Magnini, & M. de Rijke (Eds.), Accessing multilingual information repositories, 6th workshop of the cross-language evaluation forum, CLEF 2005, Vienna, Austria, September 21–23, 2005 (Revised selected papers).Google Scholar
  18. Hartrumpf, S. (2004). Question answering using sentence parsing and semantic network matching. In C. Peters, P. Clough, J. Gonzalo, G. J. F. Jones, M. Kluck, & B. Magnini (Eds.), Multilingual information access for text, speech and images, 5th workshop of the cross-language evaluation forum, CLEF 2004, Bath, September 15–17, 2004, (Revised selected papers).Google Scholar
  19. Herrera, J., Peñas, A., & Verdejo, F. (2005). Question answering pilot task at CLEF 2004. Multilingual information access for text, speech and images. CLEF 2004. Vol. 3491 of lecture notes in computer science, pp. 581–590.Google Scholar
  20. Hirschman, L., & Gaizauskas. R. (2001). Natural language question answering: the view from here. In natural language engineering, Vol. 7(4), December 2001, pp. 275–300.Google Scholar
  21. Hovy, E., Gerber, L., Hermjakob, H., Lin, C., & Ravichandran, D. (2001). Toward semantics-based answer pinpointing. In Proceedings of the DARPA human language technology conference (HLT), San Diego, CA.Google Scholar
  22. Iftene, A., Trandabat, D., Pistol, I., Moruz, A. M., Husarciuc, M., Sterpu, M., & Turliuc, C. (2009). Question answering on english and romanian languages. In C. Peters, G. di Nunzio, M. Kurimo, Th. Mandl, D. Mostefa, A. Peñas, & G. Roda (Eds.), Multilingual information access evaluation vol. i text retrieval experiments, Workshop of the cross-language evaluation forum, CLEF 2009, Corfu, Greece, September 30–October 2, 2010. Lecture notes in computer science 6241. Springer (Revised selected papers).Google Scholar
  23. Ion, R., Ştefănescu, D., Ceauşu, A., Tufiş, D., Irimia, E., & Barbu-Mititelu, V. (2009). A trainable multi-factored QA system. In C. Peters, G. di Nunzio, M. Kurimo, Th. Mandl, D. Mostefa, A. Peñas, & G. Roda (Eds.), Multilingual information access evaluation vol. I text retrieval experiments, Workshop of the cross-language evaluation forum, CLEF 2009, Corfu, Greece, September 30–October2, 2010. Lecture notes in computer science 6241. Springer (Revised selected papers).Google Scholar
  24. Jijkoun, V., & de Rijke, M. (2007). Overview of the WiQA task at CLEF 2006. In C. Peters, P. Clough, F. C. Gey, J. Karlgren, B. Magnini, D. W. Oard, M. de Rijke, & M. Stempfhuber (Eds.), Evaluation of multilingual and multi-modal information retrieval, 7th workshop of the cross-language evaluation forum, CLEF 2006, Alicante, Spain, September 20–22, 2006 (Revised selected papers).Google Scholar
  25. Jijkoun, V., Mishne, G., & de Rijke, M. (2003). The University of Amsterdam at QA@CLEF2003. In C. Peters, J. Gonzalo, M. Braschler, & M. Kluck (Eds.), Comparative evaluation of multilingual information access systems, 4th workshop of the cross-language evaluation forum, CLEF 2003, Trondheim, Norway, August 21–22, 2003 (Revised selected papers).Google Scholar
  26. Jijkoun, V., Mishne, G., de Rijke, M., Schlobach, S., Ahn, D., & Muller, H. (2004). The University of Amsterdam at QA@CLEF 2004. In C. Peters, P. Clough, J. Gonzalo, G. J. F. Jones, M. Kluck, & B. Magnini (Eds.), Multilingual information access for text, speech and images, 5th workshop of the cross-language evaluation forum, CLEF 2004, Bath, September 15–17, 2004 (Revised selected papers).Google Scholar
  27. Kelly, D., & Lin, J. (2007). Overview of the TREC 2006 ciQA task. SIGIR Forum, 41, 1.CrossRefGoogle Scholar
  28. Lamel, L., Rosset, S., Ayache, C., Mostefa, D., Turmo, J., Comas P. (2007). Question answering on speech transcriptions: The QAST evaluation in CLEF. In C. Peters, V. Jijkoun, T. Mandl, H. Müller, D. W. Oard, & A. Peñas, D. Santos (Eds.), Advances in multilingual and multimodal information retrieval, 8th workshop of the cross-language evaluation forum, CLEF 2007, Budapest, Hungary, September 19–21, 2007 (Revised selected papers).Google Scholar
  29. Laurent, D., Séguéla, P., & Nègre, S. (2005). Cross lingual question answering using QRISTAL for CLEF 2005. In C. Peters, F. C. Gey, J. Gonzalo, H. Müller, G. J. F. Jones, M. Kluck, B. Magnini, & M. de Rijke (Eds.), Accessing multilingual information repositories, 6th workshop of the cross-language evaluation forum, CLEF 2005, Vienna, Austria, September 21–23, 2005 (Revised selected papers).Google Scholar
  30. Laurent, D., Séguéla, P., & Nègre, S. (2006). Cross lingual question answer ing using QRISTAL for CLEF 2006. In C. Peters, P. Clough, F. C. Gey, J. Karlgren, B. Magnini, D. W. Oard, M. de Rijke, & M. Stempfhuber (Eds.), Evaluation of multilingual and multi-modal information retrieval, 7th workshop of the cross-language evaluation forum, CLEF 2006, Alicante, Spain, September 20–22, 2006 (Revised selected papers).Google Scholar
  31. Laurent, D., Séguéla, P., & Nègre, S. (2007). Cross lingual question answering using QRISTAL for CLEF 2007. In: C. Peters, V. Jijkoun, T. Mandl, H. Müller, D.W. Oard, A. Peñas, & D. Santos (Eds.), Advances in multilingual and multimodal information retrieval, 8th workshop of the cross-language evaluation forum, CLEF 2007, Budapest, Hungary, September 19–21, 2007 (Revised selected papers).Google Scholar
  32. Laurent, D., Séguéla, P., & Nègre, S. (2008). Cross lingual question answering using QRISTAL for CLEF 2008. In C. Peters, T. Mandl, V. Petras, A. Peñas, H. Müller, D. Oard, V. Jijkoun, & D. Santos (Eds.), Evaluating systems for multilingual and multimodal information access, 9th workshop of the cross-language evaluation forum, CLEF 2008, Aarhus, Denmark, September 17–19, 2008 (Revised selected papers).Google Scholar
  33. Lopez, V., Uren, V. S., Sabou, M., & Motta, E. (2009). Cross ontology query answering on the semantic web: An initial evaluation. In K-CAP-2009: Proceedings of the fifth international conference on knowledge capture, Redondo Beach, CA.Google Scholar
  34. Magnini, B., Negri, M., Prevete, R., & Tanev, H. (2002). Is it the right answer? Exploiting web redundancy for answer validation In Proceedings of the 40th annual meeting of the association for computational linguistics (ACL), Philadelphia. doi:10.3115/1073083.1073154.
  35. Magnini, B., Romagnoli, S., Vallin, A., Herrera, J., Peñas, A., Peinado, V., et al. (2003). The multiple language question answering track at CLEF 2003. In C. Peters, J. Gonzalo, M. Braschler, & M. Kluck (Eds.), Comparative evaluation of multilingual information access systems, 4th workshop of the cross-language evaluation forum, CLEF 2003, Trondheim, Norway, August 21–22, 2003 (Revised selected papers).Google Scholar
  36. Magnini, B., Vallin, A., Ayache, C., Erbach, G., Peñas, A., de Rijke, M., Rocha, P., Simov, K., & Sutcliffe, R. (2004). Overview of the CLEF 2004 multilingual question answering track. In C. Peters, P. Clough, J. Gonzalo, G. J. F. Jones, M. Kluck, & B. Magnini (Eds.), Multilingual information access for text, speech and images, 5th workshop of the cross-language evaluation forum, CLEF 2004, Bath, UK, September 15–17, 2004 (Revised selected papers).Google Scholar
  37. Magnini, B., Giampiccolo, D., Forner, P., Ayache, C., Jijkoun, V., Osenova, P., et al. (2006). Overview of the CLEF 2006 multilingual question answering track. In C. Peters, P. Clough, F. C. Gey, J. Karlgren, B. Magnini, D. W. Oard, M. de Rijke, & M. Stempfhuber (Eds.), Evaluation of multilingual and multi-modal information retrieval, 7th Workshop of the cross-language evaluation forum, CLEF 2006, Alicante, September 20–22, 2006 (Revised selected papers).Google Scholar
  38. Magnini, B., Speranza, M., & Kumar, V. (2009). Towards interactive question answering: an ontology-based approach. In Proceedings of the IEEE international conference on semantic computing, September 22–24, 2010, Carnegie Mellon University, Pittsburgh, PA.Google Scholar
  39. Negri, M., Tanev, H., & Magnini, B. (2003) Bridging Languages for Question Answering: DIOGENE at CLEF-2003. In C. Peters, J. Gonzalo, M. Braschler, & M. Kluck (Eds.), Comparative evaluation of multilingual information access systems, 4th workshop of the cross-language evaluation forum, CLEF 2003, Trondheim, August 21–22, 2003 (Revised selected papers).Google Scholar
  40. Noguera, E., Llopis, F., Ferrandez, A., & Escapa, A. (2007). Evaluation of open-domain question answering systems within a time constraint. In Advanced Information networking and applications workshops, AINAW ‘07. 21st international conference.Google Scholar
  41. Papineni, K., Roukos, S., Ward, T., & Zhu, W. J. (2002). BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the association for computational linguistics (ACL), Philadelphia, July 2002.Google Scholar
  42. Peñas, A., & Rodrigo, Á. (2011) A Simple Measure to Assess Non-response. In Proceedings of 49th annual meeting of the association for computational linguisticshuman language technologies (ACL-HLT 2011), Portland, Oregon, June 19–24, 2011.Google Scholar
  43. Peñas, A., Rodrigo, A., Sama, V., & Verdejo, F. (2006). Overview of the answer validation exercise 2006. In C. Peters, P. Clough, F. C. Gey, J. Karlgren, B. Magnini, D. W. Oard, M. de Rijke, & M. Stempfhuber (Eds.), Evaluation of multilingual and multi-modal information retrieval, 7th workshop of the cross-language evaluation forum, CLEF 2006, Alicante, Spain, September 20–22, 2006 (Revised selected papers).Google Scholar
  44. Peñas, A., Rodrigo, Á., & Verdejo, F. (2007). Overview of the Answer Validation Exercise 2007. In C. Peters, V. Jijkoun, T. Mandl, H. Müller, D. W. Oard, A. Peñas, V. Petras, & D. Santos, (Eds.), Advances in multilingual and multimodal information retrieval, LNCS 5152, September 2008.Google Scholar
  45. Peñas, A., Forner, P., Sutcliffe, R., Rodrigo, Á., Forascu, C., Alegria, I., et al. (2009). Overview of ResPubliQA 2009: Question answering evaluation over European legislation. In C. Peters, G. di Nunzio, M. Kurimo, Th. Mandl, D. Mostefa, A. Peñas, & G. Roda (Eds.), Multilingual information access evaluation vol. i text retrieval experiments, workshop of the cross-language evaluation forum, CLEF 2009, Corfu, Greece, September 30–October 2. Lecture Notes in computer science 6241. Springer-Verlag, 2010. (Revised selected papers).Google Scholar
  46. Peñas, A., Forner, P., Rodrigo, A., Sutcliffe, R., Forascu, & Mota, C. (2010). Overview of ResPubliQA 2010: Question answering evaluation over european legislation. In M. Braschler, D. Harman, and E. Pianta (Eds,), Notebook papers of CLEF 2010 LABs and workshops, September 22–23, 2010, Padua, Italy.Google Scholar
  47. Pérez-Iglesias, J, Garrido, G., Rodrigo, Á, Araujo, L., & Peñas, A. (2009). Information retrieval baselines for the ResPubliQA Task. In Borri, F., Nardi, A., & Peters, C. (Eds.), Cross language evaluation forum: Working notes of CLEF 2009, Corfu, Greece, September 30–October 2.Google Scholar
  48. Prager, J. (2006). Open-domain question-answering, Foundations and trends in information retrieval 1(2), 91–231. http:/dx.doi.org/10.1561/1500000001.
  49. Prager, J., Brown, E., & Coden, A. (2000). Question-answering by predictive annotation. In Proceedings of the 23rd annual international ACM SIGIR conference on research and development in information retrieval, Athens.Google Scholar
  50. Rodrigo, Á., Peñas, A., & Verdejo, F. (2008). Overview of the answer validation exercise 2008. In C. Peters, T. Mandl, V. Petras, A. Peñas, H. Müller, D. Oard, V. Jijkoun, & D. Santos (Eds.), Evaluating systems for multilingual and multimodal information access, 9th workshop of the cross-language evaluation forum, CLEF 2008, Aarhus, September 17–19, 2008 (Revised selected papers).Google Scholar
  51. Rodrigo, A., Perez, J., Peñas, A., Garrido, G., & Araujo, L. (2009). Approaching question answering by means of paragraph validation. In C. Peters, G. di Nunzio, M. Kurimo, Th. Mandl, D. Mostefa, A. Peñas, & G. Roda (Eds.), Multilingual information access evaluation vol. I text retrieval experiments, workshop of the Cross-Language Evaluation Forum, CLEF 2009, Corfu, Greece, September 30–October 2, 2010. Lecture notes in computer science 6241. Springer. Revised selected papers).Google Scholar
  52. Saias, J., & Quaresma, P. (2007). The senso question answering approach to Portuguese QA@CLEF-2007. In C. Peters, V. Jijkoun, T. Mandl, H. Müller, D. W. Oard, A. Peñas, & D. Santos (Eds.), Advances in multilingual and multimodal information retrieval, 8th workshop of the cross-language evaluation forum, CLEF 2007, Budapest, September 19–21, 2007 (Revised selected papers).Google Scholar
  53. Saias, J., & Quaresma, P. (2008) The senso question answering system at QA@CLEF 2008. In C. Peters, T. Mandl, V. Petras, A. Peñas, H. Müller, D. Oard, V. Jijkoun, & D. Santos (Eds.), Evaluating systems for multilingual and multimodal information access, 9th workshop of the cross-language evaluation forum, CLEF 2008, Aarhus, September 17–19, 2008 (Revised selected papers).Google Scholar
  54. Santos, D., & Cabral, L. M. (2009). GikiCLEF: Crosscultural issues in an international setting: Asking non-English-centered questions to Wikipedia. In Borri, F., Nardi, A., & Peters, C. (Eds.), Cross language evaluation forum: working notes of CLEF 2009, Corfu, Greece, September 30–October 2.Google Scholar
  55. Toba, H., Sari, S., Adriani, M., & Manurung, M. (2010). Contextual approach for paragraph selection in question answering task. In M. Braschler, D. Harman, & E. Pianta (Eds.), Notebook papers of CLEF 2010 LABs and workshops, September 22–23, 2010, Padua, Italy.Google Scholar
  56. Vallin, A., Magnini, B., Giampiccolo, D., Aunimo, L., Ayache, C., & Osenova, P. (2005). Overview of the CLEF 2005 multilingual question answering track. In C. Peters, F. C. Gey, J. Gonzalo, H. Müller, G. J. F. Jones, M. Kluck, B. Magnini, & M. de Rijke (Eds.), Accessing multilingual information repositories, 6th workshop of the cross-language evaluation forum, CLEF 2005, Vienna, Austria, September 21–23, 2005 (Revised selected papers).Google Scholar
  57. van Rijsbergen, K. J. (1979). Information retrieval. London: Butterworth.Google Scholar
  58. Vicedo, J. L., Izquierdo, R., Llopis, F., & Muñoz, R. (2003). Question answering in Spanish. In C. Peters, J. Gonzalo, M. Braschler, & M. Kluck (Eds.), Comparative evaluation of multilingual information access systems, 4th workshop of the cross-language evaluation forum, CLEF 2003, Trondheim, Norway, August 21–22, 2003 (Revised selected papers).Google Scholar
  59. Vicedo, J.L., Saiz, M., & Izquierdo, R. (2004). Does english help question answering in Spanish? In C. Peters, P. Clough, J. Gonzalo, G. J. F. Jones, M. Kluck, & B. Magnini (Eds.), Multilingual information access for text, speech and images, 5th workshop of the cross-language evaluation forum, CLEF 2004, Bath, UK, September 15–17, 2004 (Revised selected papers).Google Scholar
  60. Voorhees, E. M. (2000). Overview of the TREC9 question answering track. In Proceedings of the Ninth Text Retrieval Conference (TREC-9).Google Scholar
  61. Voorhees, E. M. (2002). Overview of the TREC 2002 Question answering track. In Proceedings of the eleventh text retrieval conference (TREC-11). Google Scholar
  62. Voorhees, E. M., Tice, D. M. (1999). The TREC-8 question answering track evaluation. In Proceedings of the eight text retrieval conference (TREC-8). Google Scholar
  63. Webb, N., & Webber, B. (Eds.) (2009). Special issue on interactive question answering. Journal of Natural Language Engineering, 15(1), Cambridge University Press.Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2012

Authors and Affiliations

  • Anselmo Peñas
    • 1
  • Bernardo Magnini
    • 2
  • Pamela Forner
    • 3
  • Richard Sutcliffe
    • 4
  • Álvaro Rodrigo
    • 1
  • Danilo Giampiccolo
    • 3
  1. 1.NLP & IR GroupUNEDMadridSpain
  2. 2.Fondazione Bruno Kessler (FBK-irst)TrentoItaly
  3. 3.Center for the Evaluation of Language and Communication Technologies (CELCT)TrentoItaly
  4. 4.University of LimerickLimerickIreland

Personalised recommendations