Practical Random Access to SLP-Compressed Texts

Abstract

Grammar-based compression is a popular and powerful approach to compressing repetitive texts but until recently its relatively poor time-space trade-offs during real-life construction made it impractical for truly massive datasets such as genomic databases. In a recent paper (SPIRE 2019) we showed how simple pre-processing can dramatically improve those trade-offs, and in this paper we turn our attention to one of the features that make grammar-based compression so attractive: the possibility of supporting fast random access. This is an essential primitive in many algorithms that process grammar-compressed texts without decompressing them and so many theoretical bounds have been published about it, but experimentation has lagged behind. We give a new encoding of grammars that is about as small as the practical state of the art (Maruyama et al., SPIRE 2013) but with significantly faster queries.

Appendices

A Theoretical Bounds

Charikar et al.  [7] and Rytter  [28, 29] independently showed how, given a text T of length n over an alphabet of size $\sigma$ whose smallest SLP has $g^*$ rules, in $O (n \log \sigma )$ time we can build an SLP for T with $O (g^* \log (n / g^*))$ rules and height $O (\log n)$. We can augment the non-terminals of this SLP with the sizes of their expansions to obtain an $O (g^* \log (n / g^*))$-space data structure supporting access to any $\ell$ consecutive characters of T in $O (\log n + \ell )$ time. Bille et al.  [5] showed how we can take any SLP for T with g rules, regardless of height, and build a data structure of size O(g) (measured in words of bit length $\log n$) that also supports access to any $\ell$ consecutive characters in $O (\log n + \ell )$ time, while Verbin and Yu  [35] proved we generally cannot support $O (\log ^{1 - \epsilon } n)$-time random access to T with a $\mathrm {poly} (g)$-space data structure. Belazzougui et al.  [2] showed how we can support $O (\log n / \log \log n)$-time random access to T with an $O (g \log ^\epsilon n)$-space grammar. Prezza  [27] sidestepped Verbin and Yu’s lower bound to obtain constant-time random access to T with an $O (g n^\epsilon )$-space grammar (after Belazzougui et al.  [3] achieved that tradeoff with block trees). Recently, Ganardi, Jeż and Lohrey  [15] showed how we can turn any SLP for T with g rules into an SLP for T with O(g) rules and height $O (\log n)$, thus simplifying many previous proofs.

Regarding SLPs produced with RePair, Charikar et al.  [7] showed they can be an $\varOmega (\log ^{1 / 2} n)$ factor larger than the smallest possible SLPs, and Hucke, Jeż and Lohrey  [1, 18] improved that lower bound to $\varOmega (\log n / \log \log n)$. Charikar et al. showed they are always within an $O ((n / \log n)^{2 / 3})$-factor of the smallest SLPs and this is still the best upper bound known, although Hucke  [17] showed they are within a $\log _2 3$-factor for unary strings.

B Additional experimental results

We are mainly interested in compressing human DNA but we performed experiments with other datasets to check our approach’s robustness: 11264 Salmonella genomes (salx11264) from the GenomeTrakr project  [32], and two repetitive files from the Pizza & Chili corpus⁵ (einstein.en.txt and kernel).

As can be seen from Tables 2 and 3 below and comparing Fig. 2 to Fig. 3, our results are not as good for the other datasets as for chr19x1000 but our general conclusions are supported: MTSS and OURS are about the same size and several times smaller than NAIVE; NAIVE is by far the fastest to build, with MTSS slower by almost an order of magnitude and OURS slower even than that by a factor of 4 to 7; NAIVE is also the fastest to answer queries, followed by OURS and then MTSS. Since the scale again makes it difficult to discern the height of the rightmost points, we note that NAIVE, MTSS and OURS with 8 threads use 0.53, 9.34 and 3.76 $\upmu$s for salx11264; 0.15, 6.16 and 1.84 for einstein.en.txt; and 0.53, 22.18 and 12.84 for kernel.

Fig. 2.

Average time to answer an expansion query using multiple threads.

Table 2. Statistics of our datasets: name, alphabet size, length (in bytes), number of symbols on the right-hand side of the start rule, number of rules, number of distinct expansion lengths, and height of the grammar.

Table 3. Sizes of the encodings and construction times.

Fig. 3.

Average time to answer an expansion query with expansion length 10 using multiple threads.