Abstract
The TREC 2013 Contextual Suggestion Track allowed participants to submit personalised rankings using documents either from the OpenWeb or from an archived, static Web collection, the ClueWeb12 dataset. We argue that this setting poses problems in how the performance of the participants should be compared. We analyse biases found in the process, both objective and subjective, and discuss these issues in the general framework of evaluating personalised Information Retrieval using dynamic against static datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Dean-Hall, A., Clarke, C.L.A., Kamps, J., Thomas, P.: Evaluating contextual suggestion. In: Proceedings of the Fifth International Workshop on Evaluating Information Access (EVIA 2013) (2013)
Dean-Hall, A., Clarke, C.L.A., Kamps, J., Thomas, P., Voorhees, E.: Overview of the TREC 2012 contextual suggestion track. In: Proceedings of the Twenty First Text REtrieval Conference (TREC 2012). NIST (2013)
Kelly, D.: Methods for evaluating interactive information retrieval systems with users. Foundations and Trends in Information Retrieval 3(1-2), 1–224 (2009)
Minka, T., Robertson, S.: Selection bias in the LETOR datasets. In: Proceedings of SIGIR 2008 Workshop on Learning to Rank for Information Retrieval (2008)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Bellogín, A., Samar, T., de Vries, A.P., Said, A. (2014). Challenges on Combining Open Web and Dataset Evaluation Results: The Case of the Contextual Suggestion Track. In: de Rijke, M., et al. Advances in Information Retrieval. ECIR 2014. Lecture Notes in Computer Science, vol 8416. Springer, Cham. https://doi.org/10.1007/978-3-319-06028-6_37
Download citation
DOI: https://doi.org/10.1007/978-3-319-06028-6_37
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-06027-9
Online ISBN: 978-3-319-06028-6
eBook Packages: Computer ScienceComputer Science (R0)