Advertisement

Abstract

The fourth CLEF campaign, CLEF 2003, was marked by a large increase in the number of experiments submitted by participants. More complex tasks (multilingual retrieval on documents written in eight different languages), new document sources, and a new, ninth document language, Russian, led to a wide variety of work being conducted in this campaign. This paper discusses the tracks that have been offered since the beginnings of CLEF, and that still have the highest participation, i.e. the ad-hoc and domain-specific tracks. Collectively, these tracks attracted 33 participating groups who submitted more than 400 experiments. This overview gives a brief description of the tracks and tasks, and summarizes the principal research results. A discussion of the validity of the results, in terms of the completeness of the relevance assessments used in the test collection and of the statistical significance of some of the results, is also given.

Keywords

Target Language Test Collection Relevance Assessment Topic Language Interactive Track 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Blustein, J.: IR STAT PAK, URL: http://www.csd.uwo.ca/~jamie/IRSP-overview.html
  2. 2.
    Braschler, M.: CLEF 2000 - Overview of Results. In: Peters, C. (ed.) CLEF 2000. LNCS, vol. 2069, pp. 89–101. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  3. 3.
    Braschler, M.: CLEF 2001 - Overview of Results. In: Peters, C., Braschler, M., Gonzalo, J., Kluck, M. (eds.) CLEF 2001. LNCS, vol. 2406, pp. 9–26. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  4. 4.
    Braschler, M.: CLEF 2002 – Overview of Results. In: Peters, C., Braschler, M., Gonzalo, J. (eds.) CLEF 2002. LNCS, vol. 2785, pp. 9–27. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  5. 5.
    Chen, A., Gey, F.C.: Combining Query Translation and Document Translation in Cross-Language Retrieval. In: Peters, C., Gonzalo, J., Braschler, M., Kluck, M. (eds.) CLEF 2003. LNCS, vol. 3237, pp. 108–121. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  6. 6.
    Clough, P., Sanderson, M.: The CLEF 2003 Cross Language Image Retrieval Track. In: Peters, C., Gonzalo, J., Braschler, M., Kluck, M. (eds.) CLEF 2003. LNCS, vol. 3237, pp. 581–593. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  7. 7.
    Cross-Language Evaluation Forum: Results of the CLEF 2003 Cross-Language System Evaluation Campaign. In: Working Notes for the CLEF 2003 Workshop, vol. II: Appendices. Electronic version, Available at: www.clef-campaign.org
  8. 8.
    Federico, M., Jones, G.J.F.: The CLEF 2003 Cross-Language Spoken Document Retrieval Track. In: Peters, C., Gonzalo, J., Braschler, M., Kluck, M. (eds.) CLEF 2003. LNCS, vol. 3237, pp. 646–652. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  9. 9.
    Harman, D., Braschler, M., Hess, M., Kluck, M., Peters, C., Schäuble, P., Sheridan, P.: CLIR Evaluation at TREC. In: Peters, C. (ed.) CLEF 2000. LNCS, vol. 2069, pp. 7–23. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  10. 10.
    Hull, D.A.: Using Statistical Testing in the Evaluation of Retrieval Experiments. In: Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Pittsburg, USA, pp. 329–338 (1993)Google Scholar
  11. 11.
    Kamps, J., Monz, C., de Rijke, M., Sigurbjörnsson, B.: Language-Dependent and Language-Independent Approaches to Cross-Lingual Text Retrieval. In: Peters, C., Gonzalo, J., Braschler, M., Kluck, M. (eds.) CLEF 2003. LNCS, vol. 3237, pp. 152–165. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  12. 12.
    Magnini, B., Romagnoli, S., Vallin, A., Herrera, J., Peñas, A., Peinado, V., Verdejo, M.F., de Rijke, M.: The Multiple Language Question Answering Track at CLEF 2003. In: Peters, C., Gonzalo, J., Braschler, M., Kluck, M. (eds.) CLEF 2003. LNCS, vol. 3237, pp. 471–486. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  13. 13.
    Oard, D.W., Gonzalo, J.: The CLEF 2003 Interactive Track. In: Peters, C., Gonzalo, J., Braschler, M., Kluck, M. (eds.) CLEF 2003. LNCS, vol. 3237, pp. 425–434. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  14. 14.
    Peters, C., Braschler, M.: European Research Letter: Cross-language System Evaluation: The CLEF campaigns. Journal of the American Society for Information Science and Technology 52(12), 1067–1072 (2001)CrossRefGoogle Scholar
  15. 15.
    Savoy, J.: Report on CLEF-2003 Multilingual Tracks. In: Peters, C., Gonzalo, J., Braschler, M., Kluck, M. (eds.) CLEF 2003. LNCS, vol. 3237, pp. 64–73. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  16. 16.
    Tague-Sutcliffe, J.: The Pragmatics of Information Retrieval Experimentation, Revisited. In: Readings in Information Retrieval, pp. 205–216. Morgan Kaufmann Publishers, San Francisco (1997)Google Scholar
  17. 17.
    Tague-Sutcliffe, J., Blustein, J.: A Statistical Analysis of the TREC-3 Data. In: Proc. TREC-3, NIST Special Publication 500-226, p. 385 (1994)Google Scholar
  18. 18.
    Voorhees, E., Buckley, C.: The Effect of Topic Set Size on Retrieval Experiment Error. In: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 316–323 (2002)Google Scholar
  19. 19.
    Vorhees, E., Harman, D.: Overview of the Eighth Text REtrieval Conference (TREC-8). In: Proc. TREC-8, NIST Special Publication 500-246, pp. 1–24 (1999)Google Scholar
  20. 20.
    Zobel, J.: How reliable are the results of large-scale information retrieval experiments? In: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 307–314 (1998)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Martin Braschler
    • 1
    • 2
  1. 1.Eurospider Information Technology AGZürichSwitzerland
  2. 2.Institut interfacultaire d’informatiqueUniversité de NeuchâtelNeuchâtelSwitzerland

Personalised recommendations