CLEF 2004: Ad Hoc Track Overview and Results Analysis

  • Martin Braschler
  • Giorgio M. Di Nunzio
  • Nicola Ferro
  • Carol Peters
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3491)

Abstract

We describe the objectives and organization of the CLEF 2004 ad hoc track and discuss the main characteristics of the experiments. The results are analyzed and commented and their statistical significance is investigated. The paper concludes with some observations on the impact of the CLEF campaign on the state-of-the-art in cross-language information retrieval.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Peters, C.: What happened in CLEF 2004? In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 1–9. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  2. 2.
    Cleverdon, C.: The Cranfield Tests on Index Language Devices. In: Sparck-Jones, K., Willett, P. (eds.) Readings in Information Retrieval, pp. 47–59. Morgan Kaufmann, San Francisco (1997)Google Scholar
  3. 3.
    Harman, D.: The TREC Conferences. In: Kuhlen, R., Rittberger, M. (eds.) Hypertext - Information Retrieval - Multimedia: Synergieeffekte Elektronischer Informationssysteme, Proceedings of HIM 1995, Universitätsverlag Konstanz, pp. 9–28 (1995)Google Scholar
  4. 4.
  5. 5.
    Braschler, M.: CLEF 2003 – Overview of results. In: Peters, C., Gonzalo, J., Braschler, M., Kluck, M. (eds.) CLEF 2003. LNCS, vol. 3237, pp. 44–63. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  6. 6.
    Braschler, M., Peters, C.: CLEF 2003 Methodology and Metrics. In: Peters, C., Gonzalo, J., Braschler, M., Kluck, M. (eds.) CLEF 2003. LNCS, vol. 3237, pp. 7–20. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  7. 7.
    Besançon, R., Ferre, O., Fluhr, C.: LIC2M Experiments at CLEF 2004. In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491. Springer, Heidelberg (2005)Google Scholar
  8. 8.
    Martínez-Santiago, F., García-Cumbreras, M.A., Díaz-Galiano, M.C., Ureña, L.A.: SINAI at CLEF 2004: Using Machine Translation Resources with a Mixed 2-Step RSV Merging Algorithm. In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 156–164. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  9. 9.
    Levow, G.-A., Matveeva, I.: University of Chicago at CLEF2004: Cross-language Text and Spoken Document Retrieval. In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 170–179. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  10. 10.
    Tomlinson, S.: Portuguese and Russian Retrieval with Hummingbird SearchServerTM at CLEF 2004. In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 221–232. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  11. 11.
    Di Nunzio, G.M., Ferro, N., Orio, N.: Experiments on Statistical Approaches to Compensate for Limited Linguistic Resources. In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 60–72. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  12. 12.
    Savoy, J.: Data Fusion for Effective European Monolingual Information Retrieval. In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 233–244. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  13. 13.
    Kamps, J., Adafre, S.F., de Rijke, M.: Effective Translation, Tokenization and Combination for Cross-Lingual Retrieval. In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 123–134. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  14. 14.
    Singhal, A., Buckley, C., Mitra, M.: Pivoted Document Length Normalization. In: Frei, H.P., Harman, D., Schäuble, P., Wilkinson, R. (eds.) Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1996), Zurich, Switzerland, pp. 21–29. ACM, New York (1996)CrossRefGoogle Scholar
  15. 15.
    Robertson, S.E., Walker, S., Jones, S., Hancock-Beaulieu, M.M., Gatford, M.: Okapi at TREC-3. In: Overview of the Third Text Retrieval Conference (TREC-3), pp. 109–126. NIST Special Publication 500-225 (1995)Google Scholar
  16. 16.
    Lioma, C., He, B., Plachouras, V., Ounis, I.: The University of Glasgow at CLEF 2004: French Monolingual Information Retrieval with Terrier. In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 253–259. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  17. 17.
    Serasset, G., Chevallet, J.-P.: Using Surface-Syntactic Parser and Derivation from Randomness: X-IOTA IR System used for CLIPS Mono & Bilingual Experiments for CLEF 2004. In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 38–49. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  18. 18.
    Ruiz, M.E., Srikanth, M.: UB at CLEF2004: Cross-language Information Retrieval using Statistical Language Models. In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 180–187. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  19. 19.
    Chen, A.: Cross-Language Retrieval Experiments at CLEF 2002. In: Peters, C., Braschler, M., Gonzalo, J. (eds.) CLEF 2002. LNCS, vol. 2785, pp. 5–20. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  20. 20.
    Braschler, M.: Combination Approaches for Multilingual Text Retrieval. In: Information Retrieval, vol. 7(1-2), pp. 183–204. Kluwer Academic Publishers, DordrechtGoogle Scholar
  21. 21.
    Savoy, J., Berger, P.-Y.: Selection and Merging Strategies for Multilingual Information Retrieval. In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 27–37. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  22. 22.
    Goñi-Menoyo, J.M., González, J.C., Martínez-Fernández, J.L., Villena-Román, J.: MIRACLE’s Hybrid Approach to Bilingual and Monolingual Information Retrieval. In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 188–199. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  23. 23.
    Nadeau, D., Jarmasz, M., Barrière, C., Foster, G., St-Jacques, C.: Using COTS Search Engines and Custom Query Strategies at CLEF. In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 100–109. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  24. 24.
    Hackl, R., Mandl, T., Womser-Hacker, C.: Mono- and Crosslingual Retrieval Experiments at the University of Hildesheim. In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 165–169. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  25. 25.
    Braschler, M., Peters, C.: Cross-Language Evaluation Forum: Objectives, Results, Achievements. Information Retrieval 7(1-2), 5–29Google Scholar
  26. 26.
    Gey, F.C.: Searching a Russian Document Collection using English, Chinese and Japanese Queries. In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 200–206. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  27. 27.
    Jones, G.J.F., Burke, M., Judge, J., Khasin, A., Lam-Adesina, A., Wagner, J.: Dublin City University at CLEF 2004: Experiments in Monolingual, Bilingual and Multilingual Retrieval. In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 207–220. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  28. 28.
    Hull, D.: Using Statistical Testing in the Evaluation of Retrieval Experiments. In: Korfhage, R., Rasmussen, E., Willett, P. (eds.) Proc. 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1993), pp. 329–338. ACM Press, New York (1993)CrossRefGoogle Scholar
  29. 29.
  30. 30.
    Conover, W.J.: Practical Nonparametric Statistics, 1st edn. John Wiley and Sons, New York (1971)Google Scholar
  31. 31.
    Judge, G.G., Hill, R.C., Griffiths, W.E., Lütkepohl, H., Lee, T.C.: Introduction to the Theory and Practice of Econometrics, 2nd edn. John Wiley and Sons, New York (1988)MATHGoogle Scholar
  32. 32.
    Tague-Sutcliffe, J.: The Pragmatics of Information Retrieval Experimentation, Revisited. In: Reading in Information Retrieval, pp. 205–216. Morgan Kaufmann Publishers, San Francisco (1997)Google Scholar
  33. 33.
    Voorhees, E., Buckley, C.: The Effect of Topic Set Size on Retrieval Experiment Error. In: Proc. of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 307–314 (1998)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Martin Braschler
    • 1
  • Giorgio M. Di Nunzio
    • 2
  • Nicola Ferro
    • 2
  • Carol Peters
    • 3
  1. 1.Eurospider Information Technology AGZurichSwitzerland
  2. 2.Department of Information EngineeringUniversity of PaduaItaly
  3. 3.ISTI-CNRPisaItaly

Personalised recommendations