Extreme value theory applied to document retrieval from large collections
- 70 Downloads
We consider text retrieval applications that assign query-specific relevance scores to documents drawn from particular collections. Such applications represent a primary focus of the annual Text Retrieval Conference (TREC), where the participants compare the empirical performance of different approaches. P(K), the proportion of the top K documents that are relevant, is a popular measure of retrieval effectiveness.
Participants in the TREC Very Large Corpus track have observed that when the target is a random sample from a collection, P(K) is substantially smaller than when the target is the entire collection. Hawking and Robertson (2003) confirmed this finding in a number of experimental settings. Hawking et al. (1999) posed as an open research question the cause of this phenomenon and proposed five possible explanatory hypotheses. In this paper, we present a mathematical analysis that sheds some light on these hypotheses and complements the experimental work of Hawking and Robertson (2003). We will also introduce C(L), contamination at L, the number of irrelevant documents amongst the top L relevant documents, and describe its properties.
Our analysis shows that while P(K) typically will increase with collection size, the phenomenon is not universal. That is, the asymptotic behavior of P(K) and C(L) depends on the score distributions and relative proportions of relevant and irrelevant documents in the collection.
Keywordsvery large corpus IR extreme value theory precision at K
Unable to display preview. Download preview PDF.
- Arampatzis A and van Hameren A (2001) The score-distributional threshold optimization for adaptive binary classification tasks. In: Proceedings of the 24th ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 285–293Google Scholar
- Bennett PN (2003) Using asymmetric distributions to improve text classifier probability estimates In: Proceedings of SIGIR-03, 26th ACM International Conference on Research and Development in Information Retrieval, pp. 111–118Google Scholar
- Cooper WS (1968) Expected search length: A single measure of retrieval effectiveness based on weak ordering action of retrieval systems Journal of the American Society for Information Science, 19:30–41Google Scholar
- Cox DR and Hinkley DV (1974) Theoretical Statistics. CRC Press, LondonGoogle Scholar
- David HA and Nagaraja HN (2003) Order Statistics (Third Edition), Wiley-Interscience, New YorkGoogle Scholar
- DeGroot MH and Schervish MJ (2002) Probability and Statistics (Third Edition), Addison-Wesley, New YorkGoogle Scholar
- Manmatha R, Feng F and Rath T (2001) Using models of score distributions in information retrieval. In: Proceedings of the 24th ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 267–275Google Scholar