Advertisement

Challenges on Combining Open Web and Dataset Evaluation Results: The Case of the Contextual Suggestion Track

  • Alejandro Bellogín
  • Thaer Samar
  • Arjen P. de Vries
  • Alan Said
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8416)

Abstract

The TREC 2013 Contextual Suggestion Track allowed participants to submit personalised rankings using documents either from the OpenWeb or from an archived, static Web collection, the ClueWeb12 dataset. We argue that this setting poses problems in how the performance of the participants should be compared. We analyse biases found in the process, both objective and subjective, and discuss these issues in the general framework of evaluating personalised Information Retrieval using dynamic against static datasets.

Keywords

Information Retrieval Information Retrieval System Relevance Judgement Evaluation Judgement Mean Reciprocal Rank 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Dean-Hall, A., Clarke, C.L.A., Kamps, J., Thomas, P.: Evaluating contextual suggestion. In: Proceedings of the Fifth International Workshop on Evaluating Information Access (EVIA 2013) (2013)Google Scholar
  2. 2.
    Dean-Hall, A., Clarke, C.L.A., Kamps, J., Thomas, P., Voorhees, E.: Overview of the TREC 2012 contextual suggestion track. In: Proceedings of the Twenty First Text REtrieval Conference (TREC 2012). NIST (2013)Google Scholar
  3. 3.
    Kelly, D.: Methods for evaluating interactive information retrieval systems with users. Foundations and Trends in Information Retrieval 3(1-2), 1–224 (2009)Google Scholar
  4. 4.
    Minka, T., Robertson, S.: Selection bias in the LETOR datasets. In: Proceedings of SIGIR 2008 Workshop on Learning to Rank for Information Retrieval (2008)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Alejandro Bellogín
    • 1
    • 2
  • Thaer Samar
    • 1
  • Arjen P. de Vries
    • 1
  • Alan Said
    • 1
  1. 1.Centrum Wiskunde & InformaticaAmsterdamThe Netherlands
  2. 2.Universidad Autónoma de MadridMadridSpain

Personalised recommendations