Fast Single-Pass Construction of a Half-Inverted Index
We show how a half-inverted index can be constructed twice as fast as an ordinary inverted index. As shown in a series of recent works, the half-inverted index enables very fast prefix search, which in turn is the basis for very fast processing of many other types of advanced queries. Our construction algorithm is truly single-pass in that every posting (word occurrence) is touched (read and written) only once in the whole construction by avoiding an expensive merge of the index. The algorithm has been carefully engineered, with special attention paid to cache-efficiency and disk cost. We compared our algorithm against the state-of-the-art index construction from Zettair.
Unable to display preview. Download preview PDF.
- 1.Witten, I.H., Moffat, A., Bell, T.C.: Managing gigabytes: Compressing and indexing documents and images (1999)Google Scholar
- 2.Zobel, J., Moffat, A.: Inverted files for text search engines. ACM Comput. Surv. (2006)Google Scholar
- 3.Holger Bast, I.W.: Type less, find more: fast autocompletion search with a succinct index. In: SIGIR (2006)Google Scholar
- 4.Bast, H., Weber, I.: The CompleteSearch engine: Interactive, efficient, and towards IR & DB integration. In: CIDR (2007)Google Scholar
- 5.Heinz, S., Zobel, J.: Efficient single-pass index construction for text databases. Jour. of the American Society for Information Science and Technology (2003)Google Scholar
- 6.Rogers, W., Gerald, C, Harman, D.: Space and time improvements for indexing in information retrieval. In: Proceedings of 4th Annual Symposium on Document Analysis and Information Retrieval (1995)Google Scholar
- 7.Moffat, A., Bell, T.A.H. In situ generation of compressed inverted files. Journal of the American Society for Information Science (1995)Google Scholar
- 9.Buttcher, S., Clarke, C.L.A.: Memory management strategies for single-pass index construction in text retrieval systems. Technical report, School of Computer Science, University of Waterloo, Canada (2005)Google Scholar
- 10.Heinz, S., Zobel, J.: Performance of data structure for small sets of strings. In: Proc. of the Australasian conference on Computer Science (2002)Google Scholar
- 11.Moffat, A., Zobel, J.: Self-indexing inverted files for fast text retrieval. ACM Trans. Inf. Syst. (1996)Google Scholar
- 12.Popovici, F.I., Arpaci-dusseau, A.C., Arpaci-dusseau, R.H.: Robust, portable i/o scheduling with the disk mimic. In: USENIX Annual Technical Conference (2003)Google Scholar
- 13.Middleton, C., Baeza-Yates, R.: A comparison of open source search engines (2007), http://wrg.upf.edu/WRG/dctos/Middleton-Baeza.pdf