Compressed Self-indices Supporting Conjunctive Queries on Document Collections

Abstract

We prove that a document collection, represented as a unique sequence T of n terms over a vocabulary Σ, can be represented in nH ₀(T) + o(n)(H ₀(T) + 1) bits of space, such that a conjunctive query t ₁ ∧ ⋯ ∧ t _k can be answered in O(kδloglog|Σ|) adaptive time, where δ is the instance difficulty of the query, as defined by Barbay and Kenyon in their SODA’02 paper, and H ₀(T) is the empirical entropy of order 0 of T. As a comparison, using an inverted index plus the adaptive intersection algorithm by Barbay and Kenyon takes $O(k\delta\log{\frac{n_M}{\delta}})$, where n _M is the length of the shortest and longest occurrence lists, respectively, among those of the query terms. Thus, we can replace an inverted index by a more space-efficient in-memory encoding, outperforming the query performance of inverted indices when the ratio $\frac{n_M}{\delta}$ is ω(log|Σ|).