Abstract
Collecting relevance judgments (qrels) is an especially challenging part of building an information retrieval test collection. This paper presents a novel method for creating test collections by offering a substitute for relevance judgments. Our method is based on an old idea in IR: a single information need can be represented by many query articulations. We call different articulations of a particular need query aspects. By combining the top k documents retrieved by a single system for multiple query aspects, we build judgment-free qrels whose rank ordering of IR systems correlates highly with rankings based on human relevance judgments.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Carterette, B., Allan, J.: Incremental test collections. In: CIKM 2005: Proceedings of the 14th ACM International Conference on Information and Knowledge Management, pp. 680–687. ACM, New York (2005)
Carterette, B., Allan, J., Sitaraman, R.: Minimal test collections for retrieval evaluation. In: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, WA, pp. 268–275. ACM Press, New York (2006)
Sanderson, M., Joho, H.: Forming test collections with no system pooling. In: SIGIR 2004: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 33–40. ACM, New York (2004)
Buckley, C., Voorhees, E.M.: Retrieval evaluation with incomplete information. In: SIGIR 2004: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 25–32. ACM Press, New York (2004)
Sakai, T.: Alternatives to bpref. In: SIGIR 2007: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 71–78. ACM, New York (2007)
Soboroff, I., Nicholas, C., Cahan, P.: Ranking retrieval systems without relevance judgments. In: Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New Orleans, Louisiana, United States, pp. 66–73. ACM, New York (2001)
Wu, S., Crestani, F.: Methods for ranking information retrieval systems without relevance judgments. In: SAC 2003: Proceedings of the 2003 ACM symposium on Applied computing, pp. 811–816. ACM, New York (2003)
Spoerri, A.: Using the structure of overlap between search results to rank retrieval systems without relevance judgments. Information Processing and Management 43(4), 1059–1070 (2007)
Lee, J.H.: Analyses of multiple evidence combination. SIGIR Forum 31(SI), 267–276 (1997)
Larsen, B., Ingwersen, P., Kekäläinen, J.: The polyrepresentation continuum in IR. In: IIiX: Proceedings of the 1st International Conference on Information Interaction in Context, pp. 88–96. ACM Press, New York (2006)
Ingwersen, P., Järvelin, K.: The Turn: Integration of Information Seeking and Retrieval in Context. The Information Retrieval Series. Springer, New York (2005)
Skov, M., Larsen, B., Ingwersen, P.: Inter and intra-document contexts applied in polyrepresentation for best match IR. Information Processing and Management 44(5), 1673–1683 (2008)
Belkin, N.J., Cool, C., Croft, W.B., Callan, J.P.: The effect multiple query representations on information retrieval system performance. In: SIGIR 1993: Proceedings of the 16th annual international ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 339–346. ACM, New York (1993)
Belkin, N.J., Kantor, P.B., Fox, E.A., Shaw, E.A.: Combining the evidence of multiple query representations for information retrieval. Information Processing and Management 31(3), 431–448 (1995)
Kelly, D., Fu, X.: Eliciting better information need descriptions from users of information search systems. Information Processing and Management 43(1), 30–46 (2007)
Cormack, G.V., Palmer, C.R., Clarke, C.L.A.: Efficient construction of large test collections. In: SIGIR 1998: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 282–289. ACM, New York (1998)
Robertson, S.E., Walker, S., Jones, S., Beaulieu, M.H., Gatford, M.: Okapi at TREC-3. In: Proceedings of TREC-3, the 3rd Text REtrieval Conference, NIST, pp. 109–127 (1995)
Harmon, D.K.: Overview of the Third Text Retrieval Conference (TREC-3). DIANE Publishing Company (1996)
Harmon, D.K., Voorhees, E.M.: Overview of the Seventh Text Retrieval Conference (TREC-7). DIANE Publishing Company (1996)
Harmon, D.K., Voorhees, E.M.: Overview of the Eighth Text Retrieval Conference (TREC-8). DIANE Publishing Company (1996)
Kendall, M.: Rank Correlation Methods, 3rd edn. Griffin (1990)
Voorhees, E.M.: Evaluation by highly relevant documents. In: SIGIR 2001: Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 74–82. ACM, New York (2001)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Efron, M. (2009). Using Multiple Query Aspects to Build Test Collections without Human Relevance Judgments. In: Boughanem, M., Berrut, C., Mothe, J., Soule-Dupuy, C. (eds) Advances in Information Retrieval. ECIR 2009. Lecture Notes in Computer Science, vol 5478. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-00958-7_26
Download citation
DOI: https://doi.org/10.1007/978-3-642-00958-7_26
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-00957-0
Online ISBN: 978-3-642-00958-7
eBook Packages: Computer ScienceComputer Science (R0)