Skip to main content

Practical Random Access to SLP-Compressed Texts

Part of the Lecture Notes in Computer Science book series (LNTCS,volume 12303)


Grammar-based compression is a popular and powerful approach to compressing repetitive texts but until recently its relatively poor time-space trade-offs during real-life construction made it impractical for truly massive datasets such as genomic databases. In a recent paper (SPIRE 2019) we showed how simple pre-processing can dramatically improve those trade-offs, and in this paper we turn our attention to one of the features that make grammar-based compression so attractive: the possibility of supporting fast random access. This is an essential primitive in many algorithms that process grammar-compressed texts without decompressing them and so many theoretical bounds have been published about it, but experimentation has lagged behind. We give a new encoding of grammars that is about as small as the practical state of the art (Maruyama et al., SPIRE 2013) but with significantly faster queries.

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions


  1. 1.

    RePair is probably most commonly used in natural-language processing, where it is viewed as an implementation of Gage’s  [12] byte-pair encoding and used for word segmentation in neural machine translation  [31]; we refer readers to Gallé’s  [14] recent survey for more discussion.

  2. 2.

  3. 3.

    We realized after  [13] went to press that the worst-case approximation ratios in Theorems 1 and 2 should be multiplied by the length of the sliding window, but this does not affect our approach’s correctness or practicality.

  4. 4.

  5. 5.


  1. Bannai, H., et al.: The smallest grammar problem revisited. CoRR, abs/1908.06428 (2019)

    Google Scholar 

  2. Belazzougui, D., Cording, P.H., Puglisi, S.J., Tabei, Y.: Access, rank, and select in grammar-compressed strings. In: Bansal, N., Finocchi, I. (eds.) ESA 2015. LNCS, vol. 9294, pp. 142–154. Springer, Heidelberg (2015).

    CrossRef  Google Scholar 

  3. Belazzougui, D., et al.: Queries on LZ-bounded encodings. In: 2015 Data Compression Conference, pp. 83–92. IEEE (2015)

    Google Scholar 

  4. Bille, P., Li Gørtz, I., Prezza, N.: Space-efficient re-pair compression. In: 2017 Data Compression Conference (DCC), pp. 171–180. IEEE (2017)

    Google Scholar 

  5. Bille, P., Landau, G.M., Raman, R., Sadakane, K., Satti, S.R., Weimann, O.: Random access to grammar-compressed strings and trees. SIAM J. Comput. 44(3), 513–539 (2015)

    CrossRef  MathSciNet  Google Scholar 

  6. Boucher, C., Gagie, T., Kuhnle, A., Langmead, B., Manzini, G., Mun, T.: Prefix-free parsing for building big BWTs. Algorithms Mol. Biol. 14(1), 13 (2019).

    CrossRef  MATH  Google Scholar 

  7. Charikar, M., et al.: The smallest grammar problem. IEEE Trans. Inf. Theory 51(7), 2554–2576 (2005)

    CrossRef  MathSciNet  Google Scholar 

  8. Danecek, P., et al.: The variant call format and VCFtools. Bioinformatics 27(15), 2156–2158 (2011)

    CrossRef  Google Scholar 

  9. Dinklage, P., Fischer, J., Herlez, A., Kociumaka, T., Kurpicz, F.: Practical performance of space efficient data structures for longest common extensions. In: Proceedings of the Twenty-Eighth European Symposium on Algorithms (ESA) (2020, to appear)

    Google Scholar 

  10. Esposito, E., Graf, T.M., Vigna, S.: RecSplit: minimal perfect hashing via recursive splitting. In: 2020 Proceedings of the Twenty-Second Workshop on Algorithm Engineering and Experiments (ALENEX), pp. 175–185. SIAM (2020)

    Google Scholar 

  11. Furuya, I., Takagi, T., Nakashima, Y., Inenaga, S., Bannai, H., Kida, T.: MR-RePair: grammar compression based on maximal repeats. In: Data Compression Conference. DCC 2019, Snowbird, UT, USA, 26–29 March 2019, pp. 508–517 (2019)

    Google Scholar 

  12. Gage, P.: A new algorithm for data compression. C Users J. 12(2), 23–38 (1994)

    Google Scholar 

  13. Gagie, T., I, T., Manzini, G., Navarro, G., Sakamoto, H., Takabatake, Y.: Rpair: rescaling RePair with Rsync. In: Brisaboa, N.R., Puglisi, S.J. (eds.) SPIRE 2019. LNCS, vol. 11811, pp. 35–44. Springer, Cham (2019).

    CrossRef  Google Scholar 

  14. Gallé, M.: Investigating the effectiveness of BPE: the power of shorter sequences. In: Inui, K., Jiang, J., Ng, V., Wan, X. (eds.) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, 3–7 November 2019, pp. 1375–1381. Association for Computational Linguistics (2019)

    Google Scholar 

  15. Ganardi, M., Jeż, A., Lohrey, M.: Balancing straight-line programs. In: 60th IEEE Annual Symposium on Foundations of Computer Science. FOCS 2019, Baltimore, Maryland, USA, 9–12 November 2019, pp. 1169–1183 (2019)

    Google Scholar 

  16. Gańczorz, M., Jeż, A.: Improvements on re-pair grammar compressor. In: 2017 Data Compression Conference (DCC), pp. 181–190. IEEE (2017)

    Google Scholar 

  17. Hucke, D.: Approximation ratios of RePair, LongestMatch and Greedy on unary strings. In: Brisaboa, N.R., Puglisi, S.J. (eds.) SPIRE 2019. LNCS, vol. 11811, pp. 3–15. Springer, Cham (2019).

  18. Hucke, D., Jeż, A., Lohrey, M.: Approximation ratio of RePair. CoRR, abs/1703.06061 (2017)

    Google Scholar 

  19. Kempa, D., Kociumaka, T.: String synchronizing sets: sublinear-time BWT construction and optimal LCE data structure. In: Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, pp. 756–767 (2019)

    Google Scholar 

  20. Kuhnle, A., Mun, T., Boucher, C., Gagie, T., Langmead, B., Manzini, G.: Efficient construction of a complete index for pan-genomics read alignment. In: Cowen, L.J. (ed.) RECOMB 2019. LNCS, vol. 11467, pp. 158–173. Springer, Cham (2019).

    CrossRef  Google Scholar 

  21. Kuruppu, S., Puglisi, S.J., Zobel, J.: Relative Lempel-Ziv compression of genomes for large-scale storage and retrieval. In: Chavez, E., Lonardi, S. (eds.) SPIRE 2010. LNCS, vol. 6393, pp. 201–206. Springer, Heidelberg (2010).

    CrossRef  MATH  Google Scholar 

  22. Jesper Larsson, N., Moffat, A.: Offline dictionary-based compression. In: Data Compression Conference. DCC 1999, Snowbird, Utah, USA, 29–31 March 1999, pp. 296–305 (1999)

    Google Scholar 

  23. Lohrey, M.: Algorithmics on SLP-compressed strings: a survey. Groups Complex. Cryptol. 4(2), 241–299 (2012)

    CrossRef  MathSciNet  Google Scholar 

  24. Maruyama, S., Tabei, Y., Sakamoto, H., Sadakane, K.: Fully-online grammar compression. In: Kurland, O., Lewenstein, M., Porat, E. (eds.) SPIRE 2013. LNCS, vol. 8214, pp. 218–229. Springer, Cham (2013).

    CrossRef  Google Scholar 

  25. Navarro, G.: Indexing highly repetitive string collections. CoRR, abs/2004.02781 (2020)

    Google Scholar 

  26. Ohno, T., Goto, K., Takabatake, Y., I, T., Sakamoto, H.: LZ-ABT: a practical algorithm for \(\alpha \)-balanced grammar compression. In: Iliopoulos, C., Leong, H.W., Sung, W.-K. (eds.) IWOCA 2018. LNCS, vol. 10979, pp. 323–335. Springer, Cham (2018).

    CrossRef  Google Scholar 

  27. Prezza, N.: Optimal rank and select queries on dictionary-compressed text. In: Pisanti, N., Pissis, S.P. (eds.) 30th Annual Symposium on Combinatorial Pattern Matching. CPM 2019, volume 128 of LIPIcs, Pisa, Italy, 18–20 June 2019, pp. 4:1–4:12. Schloss Dagstuhl - Leibniz-Zentrum für Informatik (2019)

    Google Scholar 

  28. Rytter, W.: Application of Lempel-Ziv factorization to the approximation of grammar-based compression. Theoret. Comput. Sci. 302(1–3), 211–222 (2003)

    CrossRef  MathSciNet  Google Scholar 

  29. Rytter, W.: Grammar compression, LZ-encodings, and string algorithms with implicit input. In: Díaz, J., Karhumäki, J., Lepistö, A., Sannella, D. (eds.) ICALP 2004. LNCS, vol. 3142, pp. 15–27. Springer, Heidelberg (2004).

    CrossRef  MATH  Google Scholar 

  30. Sakai, K., Ohno, T., Goto, K., Takabatake, Y., I, T., Sakamoto, H.: RePair in compressed space and time. In: 2019 Data Compression Conference (DCC), pp. 518–527. IEEE (2019)

    Google Scholar 

  31. Sennrich, R., Haddow, B., Birch, A.: Neural machine translation of rare words with subword units. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. ACL 2016. Volume 1: Long Papers, Berlin, Germany, 7–12 August 2016. The Association for Computer Linguistics (2016)

    Google Scholar 

  32. Stevens, E.L., et al.: The public health impact of a publically available, environmental database of microbial genomes. Front. Microbiol. 8, 808 (2017)

    CrossRef  Google Scholar 

  33. Takabatake, Y., I, T., Sakamoto, H.: A space-optimal grammar compression. In: 25th Annual European Symposium on Algorithms. ESA 2017, Vienna, Austria, 4–6 September 2017, pp. 67:1–67:15 (2017)

    Google Scholar 

  34. The 1000 Genomes Project Consortium: A global reference for human genetic variation. Nature 526, 68–74 (2015)

    Google Scholar 

  35. Verbin, E., Yu, W.: Data structure lower bounds on random access to grammar-compressed strings. In: Fischer, J., Sanders, P. (eds.) CPM 2013. LNCS, vol. 7922, pp. 247–258. Springer, Heidelberg (2013).

    CrossRef  Google Scholar 

Download references


TG was partly funded by NSERC RGPIN-2020-07185, Canada, and Basal Funds FB0001, Chile. TI, HS and YT were partly funded by JSPS KAKENHI grants 19K20213, 17H01791 and 18K18111, respectively. GM was partly funded by MIUR-PRIN grant 2017WR7SHH. GN was partly funded by Basal Funds FB0001 and Fondecyt grant 1-200038, Chile. LSB was partly funded by DFG project LO 748/10-2 (QUANT-KOMP) and received travel funds from the EU’s Horizon 2020 MSC RISE program (grant 690941).

Author information

Authors and Affiliations


Corresponding author

Correspondence to Travis Gagie .

Editor information

Editors and Affiliations


A Theoretical Bounds

Charikar et al.  [7] and Rytter  [28, 29] independently showed how, given a text T of length n over an alphabet of size \(\sigma \) whose smallest SLP has \(g^*\) rules, in \(O (n \log \sigma )\) time we can build an SLP for T with \(O (g^* \log (n / g^*))\) rules and height \(O (\log n)\). We can augment the non-terminals of this SLP with the sizes of their expansions to obtain an \(O (g^* \log (n / g^*))\)-space data structure supporting access to any \(\ell \) consecutive characters of T in \(O (\log n + \ell )\) time. Bille et al.  [5] showed how we can take any SLP for T with g rules, regardless of height, and build a data structure of size O(g) (measured in words of bit length \(\log n\)) that also supports access to any \(\ell \) consecutive characters in \(O (\log n + \ell )\) time, while Verbin and Yu  [35] proved we generally cannot support \(O (\log ^{1 - \epsilon } n)\)-time random access to T with a \(\mathrm {poly} (g)\)-space data structure. Belazzougui et al.  [2] showed how we can support \(O (\log n / \log \log n)\)-time random access to T with an \(O (g \log ^\epsilon n)\)-space grammar. Prezza  [27] sidestepped Verbin and Yu’s lower bound to obtain constant-time random access to T with an \(O (g n^\epsilon )\)-space grammar (after Belazzougui et al.  [3] achieved that tradeoff with block trees). Recently, Ganardi, Jeż and Lohrey  [15] showed how we can turn any SLP for T with g rules into an SLP for T with O(g) rules and height \(O (\log n)\), thus simplifying many previous proofs.

Regarding SLPs produced with RePair, Charikar et al.  [7] showed they can be an \(\varOmega (\log ^{1 / 2} n)\) factor larger than the smallest possible SLPs, and Hucke, Jeż and Lohrey  [1, 18] improved that lower bound to \(\varOmega (\log n / \log \log n)\). Charikar et al. showed they are always within an \(O ((n / \log n)^{2 / 3})\)-factor of the smallest SLPs and this is still the best upper bound known, although Hucke  [17] showed they are within a \(\log _2 3\)-factor for unary strings.

B Additional experimental results

We are mainly interested in compressing human DNA but we performed experiments with other datasets to check our approach’s robustness: 11264 Salmonella genomes (salx11264) from the GenomeTrakr project  [32], and two repetitive files from the Pizza & Chili corpusFootnote 5 (einstein.en.txt and kernel).

As can be seen from Tables 2 and 3 below and comparing Fig. 2 to Fig. 3, our results are not as good for the other datasets as for chr19x1000 but our general conclusions are supported: MTSS and OURS are about the same size and several times smaller than NAIVE; NAIVE is by far the fastest to build, with MTSS slower by almost an order of magnitude and OURS slower even than that by a factor of 4 to 7; NAIVE is also the fastest to answer queries, followed by OURS and then MTSS. Since the scale again makes it difficult to discern the height of the rightmost points, we note that NAIVE, MTSS and OURS with 8 threads use 0.53, 9.34 and 3.76 \(\upmu \)s for salx11264; 0.15, 6.16 and 1.84 for einstein.en.txt; and 0.53, 22.18 and 12.84 for kernel.

Fig. 2.
figure 2

Average time to answer an expansion query using multiple threads.

Table 2. Statistics of our datasets: name, alphabet size, length (in bytes), number of symbols on the right-hand side of the start rule, number of rules, number of distinct expansion lengths, and height of the grammar.
Table 3. Sizes of the encodings and construction times.
Fig. 3.
figure 3

Average time to answer an expansion query with expansion length 10 using multiple threads.

Rights and permissions

Reprints and Permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gagie, T. et al. (2020). Practical Random Access to SLP-Compressed Texts. In: Boucher, C., Thankachan, S.V. (eds) String Processing and Information Retrieval. SPIRE 2020. Lecture Notes in Computer Science(), vol 12303. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-59211-0

  • Online ISBN: 978-3-030-59212-7

  • eBook Packages: Computer ScienceComputer Science (R0)