Experiments on Adaptive Set Intersections for Text Retrieval Systems

  • Erik D. Demaine
  • Alejandro López-Ortiz
  • J. Ian Munro
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2153)


In [3] we introduced an adaptive algorithm for computing the intersection of k sorted sets within a factor of at most 8k comparisons of the information-theoretic lower bound under a model that deals with an encoding of the shortest proof of the answer. This adaptive algorithm performs better for “burstier” inputs than a straightforward worst-case optimal method. Indeed, we have shown that, subject to a reasonable measure of instance difficulty, the algorithm adapts optimally up to a constant factor. This paper explores how this algorithm behaves under actual data distributions, compared with standard algorithms. We present experiments for searching 114 megabytes of text from the World Wide Web using 5,000 actual user queries from a commercial search engine. From the experiments, it is observed that the theoretically optimal adaptive algorithm is not always the optimal in practice, given the distribution of WWW text data. We then proceed to study several improvement techniques for the standard algorithms. These techniques combine improvements suggested by the observed distribution of the data as well as the theoretical results from [3]. We perform controlled experiments on these techniques to determine which ones result in improved performance, resulting in an algorithm that outperforms existing algorithms in most cases.


Adaptive Algorithm Binary Search Standard Algorithm Stop Word Logarithmic Factor 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    R. Baeza-Yates. Efficient Text Searching. PhD thesis, Department of Computer Science, University of Waterloo, 1989.Google Scholar
  2. 2.
    Svante Carlsson, Christos Levcopoulos, and Ola Petersson. Sublinear merging and natural mergesort. Algorithmica, 9:629–648, 1993.zbMATHCrossRefMathSciNetGoogle Scholar
  3. 3.
    Erik D. Demaine, Alejandro López-Ortiz, and J. Ian Munro. Adaptive set intersections, unions, and differences. In Proceedings of the 11th Annual ACM-SIAM Symposium on Discrete Algorithms, pages 743–752, San Francisco, California, January 2000.Google Scholar
  4. 4.
    Vladimir Estivill-Castro and Derick Wood. A survey of adaptive sorting algorithms. A CM Computing Surveys, 24(4):441–476, December 1992.Google Scholar
  5. 5.
    William Frakes and Richardo Baeza-Yates. Information Retrieval. Prentice Hall, 1992.Google Scholar
  6. 6.
    F. K. Hwang. Optimal merging of 3 elements with n elements. SIAM Journal on Computing, 9(2):298–320, 1980.zbMATHCrossRefMathSciNetGoogle Scholar
  7. 7.
    F. K. Hwang and S. Lin. A simple algorithm for merging two disjoint linearly ordered sets. SIAM Journal on Computing, 1(1):31–39, 1980.CrossRefMathSciNetGoogle Scholar
  8. 8.
    Michael Lesk. “Real world” searching panel at SIGIR 1997. SIGIR Forum, 32(1), Spring 1998.Google Scholar
  9. 9.
    U. Manber and G. Myers. Suffix arrays: A new method for on-line string searchs. In Proceedings of the 1st Symposium on Discrete Algorithms, pages 319–327, 1990.Google Scholar
  10. 10.
    Alistair Moffat, Ola Petersson, and Nicholas C. Wormald. A tree-based Mergesort. Acta Informatica, 35(9):775–793, August 1998.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2001

Authors and Affiliations

  • Erik D. Demaine
    • 1
  • Alejandro López-Ortiz
    • 2
  • J. Ian Munro
    • 1
  1. 1.Department of Computer ScienceUniversity of WaterlooWaterlooCanada
  2. 2.Faculty of Computer ScienceUniversity of New BrunswickFrederictonCanada

Personalised recommendations