Adaptive Succinctness

Abstract

Representing a static set of integers S, $|S| = n$ from a finite universe $U = [1{.}{.}u]$ is a fundamental task in computer science. Our concern is to represent S in small space while supporting the operations of $\mathsf {rank}$ and $\mathsf {select}$ on S; if S is viewed as its characteristi c vector, the problem becomes that of representing a bit-vector, which is arguably the most fundamental building block of succinct data structures.

Although there is an information-theoretic lower bound of $\mathcal {B}(n, u)= \lg {u\atopwithdelims ()n}$ bits on the space needed to represent S, this applies to worst-case (random) sets S, and sets found in practical applications are compressible. We focus on the case where elements of S contain non-trivial runs of consecutive elements, one that occurs in many practical situations.

Let $\mathcal {C}_n$ denote the class of ${u\atopwithdelims ()n}$ distinct sets of $n$ elements over the universe $[1{.}{.}u]$. Let also $\mathcal {C}^{n}_{g} \subset \mathcal {C}_{n}$ contain the sets whose $n$ elements are arranged in $g \le n$ runs of $\ell _i \ge 1$ consecutive elements from U for $i=1,\ldots , g$, and let $\mathcal {C}^{n}_{g, r}\subset \mathcal {C}^{n}_{g}$ contain all sets that consist of g runs, such that $r \le g$ of them have at least 2 elements.

• We introduce new compressibility measures for sets, including:

  • $\mathcal {L}_1 = \lg {|\mathcal {C}^{n}_{g}|} = \lg {{u-n+1 \atopwithdelims ()g}} + \lg {{n-1 \atopwithdelims ()g-1}}$ and

  • $\mathcal {L}_2 = \lg {|\mathcal {C}^{n}_{g,r}|} =\lg {{u-n+1 \atopwithdelims ()g}} + \lg {{n-g-1 \atopwithdelims ()r-1}} + \lg {{g\atopwithdelims ()r}}$

  We show that $\mathcal {L}_2 \le \mathcal {L}_1 \le \mathcal {B}(n, u)$.

• We give data structures that use space close to bounds $\mathcal {L}_1$ and $\mathcal {L}_2$ and support $\mathsf {rank}$ and $\mathsf {select}$ in O(1) time.

• We provide additional measures involving entropy-coding run lengths and gaps between items, data structures to support these measures, and show experimentally that these approaches are promising for real-world datasets.

Funded by the Millennium Institute for Foundational Research on Data (IMFD).