Abstract
The Cross-Language Evaluation Forum (CLEF) is now in its fourth year of activity. We summarize the main lessons learned during this period, outline the state-of-the-art of the research reported in the CLEF experiments and discuss the contribution that this initiative has made to research and development in the multilingual information access domain. We also make proposals for future directions in system evaluation aimed at meeting emerging needs.
Article PDF
Similar content being viewed by others
References
Amati G, Carpineto C and Romano G (2003) Italian monolingual information retrieval with PROSIT. In: Peters C, Braschler M, Gonzalo J and Kluck M, Eds. Advances in Cross-Language Information Retrieval: Results of the CLEF 2002 Evaluation Campaign, LNCS 2785. Spinger Verlag, pp. 257-264.
Bertoldi N and Federico M (2003) ITC-irst at CLEF 2002: Using N-best query translations for CL-IR. In: Peters C, Braschler M, Gonzalo J and Kluck M, Eds. Advances in Cross-Language Information Retrieval: Results of the CLEF 2002 Evaluation Campaign, LNCS 2785. Spinger Verlag, pp. 49-58.
Bertoldi N and Federico M (2004) Statistical models for monolingual and bilingual information retrieval. Information Retrieval, 7:51–70.
Brand R and Brünner M (2003) Océ at CLEF 2002. In: Peters C, Braschler M, Gonzalo J and Kluck M, Eds. Advances in Cross-Language Information Retrieval: Results of the CLEF 2002 Evaluation Campaign, LNCS 2785. Spinger Verlag, pp. 59-65.
Braschler M, Krause J, Peters C and Schäuble P (1999) Cross-language information retrieval (CLIR) track overview. In: Proceedings of the Seventh Text REtrieval Conference (TREC-7). NIST Special Publication 500-242, pp. 25-32.
Braschler M and Schäuble P (2001) Experiments with the Eurospider retrieval system for CLEF 2000. In: Cross-Language Information Retrieval and Evaluation. Workshop of the Cross-Language Evaluation Forum, CLEF 2000. Revised Papers, pp. 140-148.
Braschler M (2002) CLEF 2001-Overview of results. In: Peters C, Braschler M, Gonzalo J and Kluck M, Eds. Evaluation of Cross-Language Information Retrieval Systems. Second Workshop of the Cross-Language Evaluation Forum, CLEF 2001. Revised Papers.
Braschler M (2003) CLEF 2002-Overview of results. In: Peters C, Braschler M, Gonzalo J and Kluck M, Eds. Advances in Cross-Language Information Retrieval: Results of the CLEF 2002 Evaluation Campaign, LNCS 2785. Spinger Verlag, pp. 9-27.
Braschler M (2004) Combination approaches for multilingual text retrieval. Information Retrieval, 7:181–202.
Chen A and Gey FC (2004) Multilingual information retrieval using machine translation, relevance feedback and word decompounding. Information Retrieval, 7:147–180.
Cleverdon C (1977) The Cranfield tests on index language devices. In: Sparck-Jones K and Willett P, Eds. Readings in Information Retrieval: Morgan Kaufmann, pp. 47-59.
Goldsmith JA, Higgins D and Soglasnova S (2001) Automatic language-specific stemming in information retrieval. In: Peters C, Ed. Cross-Language Information Retrieval and Evaluation. Lecture Notes in Computer Science 2069. Springer Verlag, pp. 273-283.
Gollins T and Sanderson M (2001) Sheffield university: CLEF 2000 submission-Bilingual track: German to English. In: Cross-Language Information Retrieval and Evaluation. Workshop of the Cross-Language Evaluation Forum, CLEF 2000. Revised Papers, pp. 245-252.
Grefenstette G (1998), Ed. Cross-Language Information Retrieval. Kluwer Academic Publishers.
Harman D (1995) The TREC conferences. In: Kuhlen R and Rittberger M, Eds. Hypertext, Information Retrieval, Multimedia: Synergieeffekte Elektronischer Informationssysteme, Proceedings of HIM '95. Universitätsverlag Konstanz, pp. 9-28.
Harman D, Braschler M, Hess M, Kluck M, Peters C, Schäuble P and Sheridan P (2001) CLIR Evaluation at TREC. In: Cross-Language Information Retrieval and Evaluation. Workshop of the Cross-Language Evaluation Forum, CLEF 2000. Revised Papers, pp. 7-23.
Hedlund T, Airio E, Kekustalo H, Lehtokangas R, Pirkola A and Järvelin K (2004) Dictionary-based cross-language information retrieval: Learning experience from CLEF. Information Retrieval, 7:97–117.
Hiemstra D, Kraaij W, Pohlmann R and Westerveld T (2001) Translation resources, merging strategies, and relevance feedback for cross-language information retrieval. In: Cross-Language Information Retrieval and Evaluation. Workshop of the Cross-Language Evaluation Forum, CLEF 2000. Revised Papers, pp. 102-115.
Hollink V, Kamps J, Monz C and de Rijke M (2004) Monolingual retrieval for European languages. Information Retrieval, 7: 31–50.
Jones GJF and Federico M (2003) Cross-language spoken document retrieval pilot track report. In: Peters C, E Braschler M, Gonzalo J and Kluck M, Eds. Advances in Cross-Language Information Retrieval: Results of the CLEF 2002 Evaluation Campaign, LNCS 2785. Spinger Verlag, pp. 446-457.
Kando N (2003) CLIR at NTCIR Workshop 3. In: Peters C, Braschler M, Gonzalo J and Kluck M, Eds. Advances in Cross-Language Information Retrieval: Results of the CLEF 2002 Evaluation Campaign, LNCS 2785. Spinger Verlag, pp. 485-504.
Kluck M and Gey FC (2001) The domain-specific task of CLEF-Specific evaluation strategies in cross-language information retrieval. In: Peters C, Eds. Cross-Language Information Retrieval and Evaluation. Lecture Notes in Computer Science 2069. Springer Verlag, pp. 48-56.
Lam-Adesina AM and Jones GJF (2003) Exeter at CLEF 2002: Experiments with machine translation for monolingual and bilingual retrieval. In: Peters C, Braschler M, Gonzalo J and Kluck M, Eds. Advances in Cross-Language Information Retrieval: Results of the CLEF 2002 Evaluation Campaign, LNCS 2785. Spinger Verlag, pp. 127-146.
Mandl T and Womser-Hacker C (2003) Linguistic and statistical analysis of the CLEF topics. In: Peters C, Braschler M, Gonzalo J and Kluck M, Eds. Advances in Cross-Language Information Retrieval: Results of the CLEF 2002 Evaluation Campaign, LNCS 2785. Spinger Verlag, pp. 505-511.
McNamee P, Mayfield J and Piatko C (2001) A language-independent approach to European text retrieval. In: Cross-Language Information Retrieval and Evaluation. Workshop of the Cross-Language Evaluation Forum, CLEF 2000. Revised Papers, pp. 129-139.
McNamee P and Mayfield J (2004) Character N-gram tokenization for European text retrieval. Information Retrieval, 7: 71–95.
Nie J-Y, Simard M and Foster G (2001) Multilingual information retrieval based on parallel texts from the Web. In: Cross-Language Information Retrieval and Evaluation. Workshop of the Cross-Language Evaluation Forum, CLEF 2000. Revised Papers, pp. 188-201.
Oard DW, Gonzalo J, Sanderson M, López-Ostenero F and Wang J (2004) Interactive cross-language document selection. Information Retrieval, 7:203–226.
Ogden B and Du B (2000) Can monolingual users create good multilingual queries without machine translation? In: Peters C, Ed. First Results of the CLEF 2000 Cross-Language Text Retrieval System Evaluation Campaign. Working Notes for the CLEF 2000 Workshop, ERCIM-00-W01, pp. 133-134.
Pieters D (2002) The Languages of the European Union. Publication of the European Commission, IBSN 90-807420-3-1. Available at http://europa.eu.int/futurum/documents/offtext/espdiscuss10 en.pdf.
Sahlgren M and Karlgren J (2002) Vector-based semantic analysis using random indexing for cross-lingual query expansion. In: Peters C, Braschler M, Gonzalo J and Kluck M, Eds. Evaluation of Cross-Language Information Retrieval Systems. Second Workshop of the Cross-Language Evaluation Forum, CLEF 2001. Revised Papers, pp. 169-176.
Sahlgren M, Karlgren J, Cöster R and Järvinen T (2003) SICS at CLEF 2002: Automatic query expansion using random indexing. In: Peters C, Braschler M, Gonzalo J and Kluck M, Eds. Advances in Cross-Language Information Retrieval: Results of the CLEF 2002 Evaluation Campaign, LNCS 2785. Spinger Verlag, pp. 311-320.
Savoy J (2004) Combining multiple strategies for effective monolingual and cross-language retrieval. Information Retrieval, 7:119–146.
Schäuble P and Sheridan P (1998) Cross-language information retrieval (CLIR) track overview. In: Proceedings of the Seventh Text Retrieval Conference (TREC-7). NIST Special Publication 500-422, pp. 25-32.
Smeaton AF and Harman D (1977) The TREC experiments and their impact on Europe. Journal of Information Science, 23:169–174.
Salton G (1970) Automatic processing of foreign language documents. Journal of the American Society for Information Science, 21(3):187–194.
Voorhees E (2002) The Philosophy of Information Retrieval Evaluation. In: Peters C, Braschler M, Gonzalo J and Kluck M, Eds. Evaluation of Cross-Language Information Retrieval Systems. Second Workshop of the Cross-Language Evaluation Forum, CLEF 2001. Revised Papers, pp. 355-370.
Voorhees E and Buckley C (2002) The effect of topic set size on retrieval experiment error. In: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 316-323.
Womser-Hacker C (2002) Multilingual Topic Generation within the CLEF 2001 experiments. In: Peters C, Braschler M, Gonzalo J and Kluck M, Eds. Evaluation of Cross-Language Information Retrieval Systems. Second Workshop of the Cross-Language Evaluation Forum, CLEF 2001. Revised Papers, pp. 389-393.
Zobel J (1998) How reliable are the results of large-scale information retrieval experiments? In: Proceedings of the 21st Annual International ACMSIGIR Conference on Research and Development in Information Retrieval.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Braschler, M., Peters, C. Cross-Language Evaluation Forum: Objectives, Results, Achievements. Information Retrieval 7, 7–31 (2004). https://doi.org/10.1023/B:INRT.0000009438.69013.fa
Issue Date:
DOI: https://doi.org/10.1023/B:INRT.0000009438.69013.fa