Chapter

Advances in Database Technology - EDBT 2006

Volume 3896 of the series Lecture Notes in Computer Science pp 313-330

Indexing Shared Content in Information Retrieval Systems

  • Andrei Z. BroderAffiliated withYahoo! Inc
  • , Nadav EironAffiliated withGoogle Inc
  • , Marcus FontouraAffiliated withYahoo! Inc
  • , Michael HerscoviciAffiliated withIBM Haifa Research Lab
  • , Ronny LempelAffiliated withIBM Haifa Research Lab
  • , John McPhersonAffiliated withIBM Silicon Valley Lab
  • , Runping QiAffiliated withYahoo! Inc
  • , Eugene ShekitaAffiliated withIBM Almaden Research Center

* Final gross prices may vary according to local VAT.

Get Access

Abstract

Modern document collections often contain groups of documents with overlapping or shared content. However, most information retrieval systems process each document separately, causing shared content to be indexed multiple times. In this paper, we describe a new document representation model where related documents are organized as a tree, allowing shared content to be indexed just once. We show how this representation model can be encoded in an inverted index and we describe algorithms for evaluating free-text queries based on this encoding. We also show how our representation model applies to web, email, and newsgroup search. Finally, we present experimental results showing that our methods can provide a significant reduction in the size of an inverted index as well as in the time to build and query it.