Distribution-Aware Compressed Full-Text Indexes

  • Paolo Ferragina
  • Jouni Sirén
  • Rossano Venturini
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6942)


In this paper we address the problem of building a compressed self-index that, given a distribution for the pattern queries and a bound on the space occupancy, minimizes the expected query-time within that index-space bound. We solve this problem by exploiting a reduction to the problem of finding a minimum weight K-link path in a particular Directed Acyclic Graph. Interestingly enough, our solution is independent of the underlying compressed index in use. Our experiments compare this optimal strategy with several other standard approaches, showing its effectiveness in practice.


Direct Acyclic Graph Extra Space Space Occupancy Rank Query Optimal Sampling Strategy 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aggarwal, A., Schieber, B., Tokuyama, T.: Finding a minimum-weight k-link path graphs with the concae monge property and applications. Discrete & Computational Geometry 12, 263–280 (1994)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Burrows, M., Wheeler, D.: A block sorting lossless data compression algorithm. Technical Report 124, Digital Equipment Corporation (1994)Google Scholar
  3. 3.
    Ferragina, P., González, R., Navarro, G., Venturini, R.: Compressed text indexes: From theory to practice. ACM Journal of Experimental Algorithmics, 13 (2008)Google Scholar
  4. 4.
    Ferragina, P., Manzini, G.: Indexing compressed text. J. ACM 52(4), 552–581 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Ferragina, P., Manzini, G.: On compressing the textual web. In: WSDM, pp. 391–400 (2010)Google Scholar
  6. 6.
    Grossi, R., Gupta, A., Vitter, J.: High-order entropy-compressed text indexes. In: Proc. 14th ACM-SIAM Symposium on Discrete Algorithms (SODA), pp. 841–850 (2003)Google Scholar
  7. 7.
    Grossi, R., Vitter, J.: Compressed suffix arrays and suffix trees with applications to text indexing and string matching. In: Proc. of the 32nd ACM Symposium on Theory of Computing, pp. 397–406 (2000)Google Scholar
  8. 8.
    Hagerup, T., Tholey, T.: Efficient minimal perfect hashing in nearly minimal space. In: Ferreira, A., Reichel, H. (eds.) STACS 2001. LNCS, vol. 2010, pp. 317–326. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  9. 9.
    Hirschberg, D.S., Larmore, L.L.: The least weight subsequence problem. SIAM Journal on Computing 16(4), 628–638 (1987)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Manzini, G.: An analysis of the Burrows-Wheeler transform. Journal of the ACM 48(3), 407–430 (2001)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Navarro, G., Mäkinen, V.: Compressed full-text indexes. ACM Computing Surveys 39(1) (2007)Google Scholar
  12. 12.
    Sadakane, K.: New text indexing functionalities of the compressed suffix arrays. J. Algorithms 48(2), 294–313 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Schieber, B.: Computing a minimum weight k-link path in graphs with the concave monge property. J. Algorithms 29(2), 204–222 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Silvestri, F.: Mining query logs: Turning search usage data into knowledge. Foundations and Trends in Information Retrieval 4(1-2), 1–174 (2010)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Paolo Ferragina
    • 1
  • Jouni Sirén
    • 2
  • Rossano Venturini
    • 3
  1. 1.Dept. of Computer ScienceUniv. of PisaItaly
  2. 2.Dept. of Computer ScienceUniv. of HelsinkiItaly
  3. 3.ISTI-CNRPisaItaly

Personalised recommendations