GikiCLEF: Expectations and Lessons Learned

  • Diana Santos
  • Luís Miguel Cabral
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6241)


This overview paper is devoted to a critical assessment of GikiCLEF 2009, an evaluation contest specifically designed to expose and investigate cultural and linguistic issues in Wikipedia search, with eight participant systems and 17 runs. After providing a maximally short but self contained overview of the GikiCLEF task and participation, we present the open source SIGA system, and discuss, for each of the main guiding ideas, the resulting successes or shortcomings, concluding with further work and still unanswered questions.


Participant System Crosscultural Issue Linguistic Issue List Question Geographic Information Retrieval 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Santos, D., Cardoso, N., Carvalho, P., Dornescu, I., Hartrumpf, S., Leveling, J., Skalban, Y.: GikiP at GeoCLEF 2008: Joining GIR and QA forces for querying Wikipedia. In: Peters, C., Deselaers, T., Ferro, N., Gonzalo, J., Jones, G.J.F., Kurimo, M., Mandl, T., Peñas, A., Petras, V. (eds.) Evaluating Systems for Multilingual and Multimodal Information Access. LNCS, vol. 5706, pp. 894–905. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  2. 2.
    Santos, D., Rocha, P.: The key to the first CLEF in Portuguese: Topics, questions and answers in CHAVE. In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 821–832. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  3. 3.
    Santos, D., Cardoso, N.: Portuguese at CLEF 2005: Reflections and Challenges. In: Peters, C. (ed.) CLEF 2005. LNCS, vol. 4022, pp. 1007–1010. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  4. 4.
    Santos, D., Costa, L.: QolA: fostering collaboration within QA. In: Peters, C., Clough, P., Gey, F.C., Karlgren, J., Magnini, B., Oard, D.W., de Rijke, M., Stempfhuber, M. (eds.) CLEF 2006. LNCS, vol. 4730, pp. 569–578. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  5. 5.
    Zobel, J.: How Reliable Are the Results of Large-Scale Information Retrieval Experiments? In: SIGIR’98: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 307–314. ACM, New York (1998)CrossRefGoogle Scholar
  6. 6.
    Voorhees, E.M., Buckley, C.: The effect of topic set size on retrieval experiment error. In: Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 316–323 (2002)Google Scholar
  7. 7.
    Ferro, N., Harman, D.: CLEF 2009: Grid@CLEF Pilot Track Overview. In: Peters, C., et al. (eds.) CLEF 2009 Workshop, Part I. LNCS, vol. 6241, pp. 553–566. Springer, Heidelberg (2010)Google Scholar
  8. 8.
    Santos, D., Cardoso, N.: GikiP: Evaluating geographical answers from Wikipedia. In: Proceedings of the 5th Workshop on Geographic Information Retrieval (GIR 2008), Napa Valley, CA, USA, pp. 59–60 (2008)Google Scholar
  9. 9.
    Santos, D., Cabral, L.M.: GikiCLEF: Crosscultural issues in an international setting: asking non-English-centered questions to Wikipedia. In: Borri, F., Nardi, A., Peters, C. (eds.) Cross Language Evaluation Forum: Working notes for CLEF (2009)Google Scholar
  10. 10.
    Dussin, M., Ferro, N.: Direct: applying the dikw hierarchy to large-scale evaluation campaigns. In: Larsen, R.L., Paepcke, A., Borbinha, J.L., Naaman, M. (eds.) Proceedings of the 8th ACM/IEEE-CS joint conference on Digital libraries, pp. 424–424. ACM, New YorkGoogle Scholar
  11. 11.
    Lalmas, M., Piwowarski, B.: INEX 2006 relevance assessment guide. In: INEX 2006 Workshop Pre-Proceedings, pp. 389–395 (2006)Google Scholar
  12. 12.
    Denoyer, L., Gallinari, P.: The Wikipedia XML corpus. ACM SIGIR Forum 40, 272–367 (2006)CrossRefGoogle Scholar
  13. 13.
    Cardoso, N.: GikiCLEF topics and Wikipedia articles: did it blend? In: CLEF2009 Workshop, Corfu, Greece, September 30 - October 2 (2009)Google Scholar
  14. 14.
    Cardoso, N., Batista, D., Lopez-Pellicer, F., Silva, M.J.: Where in the Wikipedia is that answer? The XLDB at the GikiCLEF 2009 task. In: Borri, F., Nardi, A., Peters, C. (eds.) Cross Language Evaluation Forum CLEF 2009 Workshop (2009)Google Scholar
  15. 15.
    Santos, D., Cardoso, N.: Portuguese at CLEF. In: Peters, C., Gey, F.C., Gonzalo, J., Müller, H., Jones, G.J.F., Kluck, M., Magnini, B., de Rijke, M., Giampiccolo, D. (eds.) CLEF 2005. LNCS, vol. 4022, pp. 1007–1010. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  16. 16.
    Dornescu, I.: EQUAL Encyclopaedic QA for Lists. In: CLEF2009 Workshop, Corfu, Greece (2009)Google Scholar
  17. 17.
    Larson, R.R.: Interactive probabilistic search for GikiCLEF. In: Borri, F., Nardi, A., Peters, C. (eds.) Cross Language Evaluation Forum: Working notes for CLEF 2009, Corfu, Greece (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Diana Santos
    • 1
  • Luís Miguel Cabral
    • 1
  1. 1.Linguateca, Oslo node, SINTEF ICTNorway

Personalised recommendations