Asia Information Retrieval Symposium

Information Retrieval Technology pp 15-28 | Cite as

Access Time Tradeoffs in Archive Compression

  • Matthias Petri
  • Alistair Moffat
  • P. C. Nagesh
  • Anthony Wirth
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9460)


Web archives, query and proxy logs, and so on, can all be very large and highly repetitive; and are accessed only sporadically and partially, rather than continually and holistically. This type of data is ideal for compression-based archiving, provided that random-access to small fragments of the original data can be achieved without needing to decompress everything. The recent RLZ (relative Lempel Ziv) compression approach uses a semi-static model extracted from the text to be compressed, together with a greedy factorization of the whole text encoded using static integer codes. Here we demonstrate more precisely than before the scenarios in which RLZ excels. We contrast RLZ with alternatives based on block-based adaptive methods, including approaches that “prime” the encoding for each block, and measure a range of implementation options using both hard-disk (HDD) and solid-state disk (SSD) drives. For HDD, the dominant factor affecting access speed is the compression rate achieved, even when this involves larger dictionaries and larger blocks. When the data is on SSD the same effects are present, but not as markedly, and more complex trade-offs apply.


  1. 1.
    Bergman, A., Zohar, E.: Compressing Yahoo mail. In: Proceedings of the DCC, pp. 223–232 (2015)Google Scholar
  2. 2.
    Ferrada, H., Gagie, T., Gog, S., Puglisi, S.J.: Relative Lempel-Ziv with constant-time random access. In: Moura, E., Crochemore, M. (eds.) SPIRE 2014. LNCS, vol. 8799, pp. 13–17. Springer, Heidelberg (2014)Google Scholar
  3. 3.
    Fiala, E.R., Greene, D.H.: Data compression with finite windows. Commun. ACM 32(4), 490–505 (1989)CrossRefGoogle Scholar
  4. 4.
    Gog, S., Beller, T., Moffat, A., Petri, M.: From theory to practice: plug and play with succinct data structures. In: Gudmundsson, J., Katajainen, J. (eds.) sea 2014. LNCS, vol. 8504, pp. 326–337. Springer, Heidelberg (2014)Google Scholar
  5. 5.
    Hoobin, C., Puglisi, S.J., Zobel, J.: Relative Lempel-Ziv factorization for efficient storage and retrieval of web collections. PVLDB 5(3), 265–273 (2011)Google Scholar
  6. 6.
    Moffat, A., Turpin, A.: Compression and Coding Algorithms. Kluwer, Boston (2002)CrossRefGoogle Scholar
  7. 7.
    Tong, J., Wirth, A., Zobel, J.: Principled dictionary pruning for low-memory corpus compression. In: Proceedings of the SIGIR, pp. 283–292 (2014)Google Scholar
  8. 8.
    Webber, W., Moffat, A.: In search of reliable retrieval experiments. In: Proceedings of the 10th Australasian Document Computing Symposium, pp. 26–33 (2005)Google Scholar
  9. 9.
    Williams, H.E., Zobel, J.: Compressing integers for fast file access. Comput. J. 42(3), 193–201 (1999)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Matthias Petri
    • 1
  • Alistair Moffat
    • 1
  • P. C. Nagesh
    • 1
  • Anthony Wirth
    • 1
  1. 1.Department of Computing and Information SystemsThe University of MelbourneVictoriaAustralia

Personalised recommendations