Compressed Prefix Sums

Abstract

We consider the prefix sums problem: given a (static) sequence of positive integers $\vec{x} = (x_1, \ldots, x_n)$, such that $\sum_{i=1}^n x_i = m$, we wish to support the operation ${\sf sum}(\vec{x},j)$, which returns $\sum_{i=1}^{j} x_i$. Our interest is in minimising the space required for storing $\vec{x}$, where ‘minimal space’ is defined according to some compressibility criteria, while supporting sum as rapidly as possible.

There are two main compressibility criteria: (a) the succinct space bound, $B(m, n) = \lceil \log_2 {{m-1}\choose{n-1}} \rceil$ bits, applies to any sequence $\vec{x}$ whose elements add up to m; (b) data-aware measures, which depend on the values in $\vec{x}$, and can be lower than the succinct bound for some sequences. Appropriate data-aware measures have been studied extensively in the information retrieval (IR) community [17].

We demonstrate a close connection between the data-aware measure that is the best in practice for an important IR application and the succinct bound. We give theoretical solutions that use space close to other data-aware compressibility measures (often within o(n) bits), and support sum in doubly-logarithmic (or better) time, and experimental evaluations of practical variants thereof.

A bit-vector is a data structure that supports ‘rank/select’ on a bit-string, and is fundamental to succinct and compressed data structures. We describe a new bit-vector that is robust and efficient.

Delpratt is supported by PPARC e-Science Studentship PPA/S/E/2003/03749.