Experiments with Document Archive Size Detection

  • Shengli Wu
  • Forbes Gibb
  • Fabio Crestani
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2633)


The size of a document archive is a very important parameter for resource selection in distributed information retrieval systems. In this paper, we present a method for automatically detecting the size (i.e. number of documents) of a document archive, in case the archive itself does not provided such information. In addition, a method for detecting the incremental change of the archive size is also presented, which can be useful for deciding if a resource description has become obsolete and needs to be regenerated. An experimental evaluation of these methods shows that they provide quite accurate information.


Language Model Financial Time Wall Street Journal Incremental Change Information Retrieval System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    K. Bharat and A. Z. Broder. A Technique for Measuring the Relative Size and Overlap of Public Web Search Engines. In Proceedings of 7th WWW Conference, Brisbane, Australia, April 1998.Google Scholar
  2. [2]
    J. K. Callan, M. Connell, and A. Du. Automatic Discovery of Language Models for Text Databases. In Proceedings of ACM SIGMOD International Conference, pages 479–490, Philadelphia, USA, May 1999.Google Scholar
  3. [3]
    J. K. Callan, Z. Lu, and W. Croft. Searching Distributed Collections with Inference Networks. In Proceedings of the 18th Annual International ACM SIGIR Conference, pages 21–28, Seattle, USA, June 1995.Google Scholar
  4. [4]
    Norbert Fuhr. A Decision-Theoretic Approach to Database Selection in Networked IR. ACM Transaction on Information Systems, 17(3):229–249, 1999.CrossRefGoogle Scholar
  5. [5]
    L. Gravano and H. García-Molina. Generalizing GlOSS to Vector-Space Database and Broker Hierarchies. In Proceedings of 21st VLDB International Conference, pages 78–89, Zürich, Switzerland, 1995.Google Scholar
  6. [6]
    S. Lawrence and C. L. Giles. Searching the World Wide Web. Science, 280(3):98–100, April 1998.CrossRefGoogle Scholar
  7. [7]
    K. Liu, C. Yu, and W. Meng. Discovering the Representative of a Search Engine. In Proceedings of the ACM CIKM International Conference, pages 652–654, Mclean, Virginia, USA, November 2002.Google Scholar
  8. [8]
    P. Ogilvie and J. Callan. Experiments using the Lemur Toolkit. In Proceedings of the 2001 Text Retrieval Conference (TREC), pages 103–108, Gaithersburg, Maryland, USA, November 2001.Google Scholar
  9. [9]
    S. Wu and F. Crestani. Multi-objective Resource Selection in Distributed Information Retrieval. In Proceedings of IPMU02, International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems, pages 1171–1178, Annecy, France, July 2002.Google Scholar
  10. [10]
    B. Yuwono and D. Lee. Server Ranking for Distributed Test Retrieval Systems on the Internet. In Proceedings of the Fifth International Conference on Database Systems for Advanced Application, pages 41–50, Melbourne, Australia, April 1997.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • Shengli Wu
    • 1
  • Forbes Gibb
    • 1
  • Fabio Crestani
    • 1
  1. 1.Department of Computer and Information SciencesUniversity of StrathclydeGlasgow, ScotlandUK

Personalised recommendations