Distribution-Aware Compressed Full-Text Indexes

  • Paolo Ferragina
  • Jouni Sirén
  • Rossano Venturini
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6942)


In this paper we address the problem of building a compressed self-index that, given a distribution for the pattern queries and a bound on the space occupancy, minimizes the expected query-time within that index-space bound. We solve this problem by exploiting a reduction to the problem of finding a minimum weight K-link path in a particular Directed Acyclic Graph. Interestingly enough, our solution is independent of the underlying compressed index in use. Our experiments compare this optimal strategy with several other standard approaches, showing its effectiveness in practice.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aggarwal, A., Schieber, B., Tokuyama, T.: Finding a minimum-weight k-link path graphs with the concae monge property and applications. Discrete & Computational Geometry 12, 263–280 (1994)MathSciNetCrossRefMATHGoogle Scholar
  2. 2.
    Burrows, M., Wheeler, D.: A block sorting lossless data compression algorithm. Technical Report 124, Digital Equipment Corporation (1994)Google Scholar
  3. 3.
    Ferragina, P., González, R., Navarro, G., Venturini, R.: Compressed text indexes: From theory to practice. ACM Journal of Experimental Algorithmics, 13 (2008)Google Scholar
  4. 4.
    Ferragina, P., Manzini, G.: Indexing compressed text. J. ACM 52(4), 552–581 (2005)MathSciNetCrossRefMATHGoogle Scholar
  5. 5.
    Ferragina, P., Manzini, G.: On compressing the textual web. In: WSDM, pp. 391–400 (2010)Google Scholar
  6. 6.
    Grossi, R., Gupta, A., Vitter, J.: High-order entropy-compressed text indexes. In: Proc. 14th ACM-SIAM Symposium on Discrete Algorithms (SODA), pp. 841–850 (2003)Google Scholar
  7. 7.
    Grossi, R., Vitter, J.: Compressed suffix arrays and suffix trees with applications to text indexing and string matching. In: Proc. of the 32nd ACM Symposium on Theory of Computing, pp. 397–406 (2000)Google Scholar
  8. 8.
    Hagerup, T., Tholey, T.: Efficient minimal perfect hashing in nearly minimal space. In: Ferreira, A., Reichel, H. (eds.) STACS 2001. LNCS, vol. 2010, pp. 317–326. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  9. 9.
    Hirschberg, D.S., Larmore, L.L.: The least weight subsequence problem. SIAM Journal on Computing 16(4), 628–638 (1987)MathSciNetCrossRefMATHGoogle Scholar
  10. 10.
    Manzini, G.: An analysis of the Burrows-Wheeler transform. Journal of the ACM 48(3), 407–430 (2001)MathSciNetCrossRefMATHGoogle Scholar
  11. 11.
    Navarro, G., Mäkinen, V.: Compressed full-text indexes. ACM Computing Surveys 39(1) (2007)Google Scholar
  12. 12.
    Sadakane, K.: New text indexing functionalities of the compressed suffix arrays. J. Algorithms 48(2), 294–313 (2003)MathSciNetCrossRefMATHGoogle Scholar
  13. 13.
    Schieber, B.: Computing a minimum weight k-link path in graphs with the concave monge property. J. Algorithms 29(2), 204–222 (1998)MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Silvestri, F.: Mining query logs: Turning search usage data into knowledge. Foundations and Trends in Information Retrieval 4(1-2), 1–174 (2010)CrossRefMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Paolo Ferragina
    • 1
  • Jouni Sirén
    • 2
  • Rossano Venturini
    • 3
  1. 1.Dept. of Computer ScienceUniv. of PisaItaly
  2. 2.Dept. of Computer ScienceUniv. of HelsinkiItaly
  3. 3.ISTI-CNRPisaItaly

Personalised recommendations