Document Identifier Reassignment Through Dimensionality Reduction

  • Roi Blanco
  • Álvaro Barreiro
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3408)

Abstract

Most modern retrieval systems use compressed Inverted Files (IF) for indexing. Recent works demonstrated that it is possible to reduce IF sizes by reassigning the document identifiers of the original collection, as it lowers the average distance between documents related to a single term. Variable-bit encoding schemes can exploit the average gap reduction and decrease the total amount of bits per document pointer. However, approximations developed so far requires great amounts of time or use an uncontrolled memory size. This paper presents an efficient solution to the reassignment problem consisting in reducing the input data dimensionality using a SVD transformation. We tested this approximation with the Greedy-NN TSP algorithm and one more efficient variant based on dividing the original problem in sub-problems. We present experimental tests and performance results in two TREC collections, obtaining good compression ratios with low running times. We also show experimental results about the tradeoff between dimensionality reduction and compression, and time performance.

Keywords

Document identifier reassignment SVD indexing compression 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Bartell, B.T., Cottrel, G.W., Belew, R.K.: Latent Semantic Indexing is an optimal special case of Multidimensional Scaling. In: Proceeding of the 15th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 161–167 (1992)Google Scholar
  2. 2.
    Blandford, D., Blelloch, G.: Index compression through document reordering. In: Proceedings of the IEEE Data Compression Conference (DCC 2002), pp. 342–351 (2002)Google Scholar
  3. 3.
    Deerwester, S., Dumais, S.T., Furnas, G.W., Landauer, T.K., Harshman, R.: Indexing by Latent Semantic Analysis. Journal of the American Society for Information Science 41(6), 391–407 (1990)CrossRefGoogle Scholar
  4. 4.
    Dumais, S.T.: Latent Semantic Indexing (LSI): TREC-3 Report. In: Proceedings of the Third Text REtrieval Conference (TREC-3), NIST Special Publication 500-225 (November 1994)Google Scholar
  5. 5.
    Managing Gigabytes, http://www.cs.mu.oz.au/mg/
  6. 6.
    MG4J (Managing Gigabytes for Java), http://mg4j.dsi.unimi.it/
  7. 7.
    Moffat, A., Turpin, A.: Compression and Coding Algorithms. Kluwer, Dordrecht (2002)Google Scholar
  8. 8.
    Rivest, R.: RFC 1321: The md5 algorithmGoogle Scholar
  9. 9.
    Shieh, W.-Y., Chen, T.-F., Shann, J.J.-J., Chung, C.-P.: Inverted file compression through document identifier reassignment. Information Processing and Management 39(1), 117–131 (2003)MATHCrossRefGoogle Scholar
  10. 10.
    Silvestri, F., Orlando, S., Perego, R.: Assigning identifiers to documents to enhance the clustering property of fulltext indexes. In: Proceeding of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 305–312 (2004)Google Scholar
  11. 11.
  12. 12.
    Witten, I.H., Moffat, A., Bell, T.C.: Managing Gigabytes - Compressing and Indexing Documents and Images, 2nd edn. Morgan Kaufmann Publishing, San Francisco (1999)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Roi Blanco
    • 1
  • Álvaro Barreiro
    • 1
  1. 1.AILab. Computer Science DepartmentUniversity of CorunnaSpain

Personalised recommendations