A Two-Phase Sampling Technique to Improve the Accuracy of Text Similarities in the Categorisation of Hidden Web Databases

  • Yih-Ling Hedley
  • Muhammad Younas
  • Anne James
  • Mark Sanderson
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3306)


The larger amount of high quality and specialised information on the Web is stored in document databases, which is not indexed by general-purpose search engines such as Google and Yahoo. Such information is dynamically generated as a result of submitting queries to databases – which are referred to as Hidden Web databases. This paper presents a Two-Phase Sampling (2PS) technique that detects Web page templates from the randomly sampled documents of a database. It generates terms and frequencies that summarise the database content with improved accuracy. We then utilise such statistics to improve the accuracy of text similarity computation in categorisation. Experimental results show that 2PS effectively eliminates terms contained in Web page templates, and generates terms and frequencies with improved accuracy. We also demonstrate that 2PS improves the accuracy of text similarity computation required in the process of database categorisation.


Content Summary Document Frequency Relevant Term Text Similarity Text Segment 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bergman, M.K.: The Deep Web: Surfacing Hidden Value. Appeared in The Journal of Electronic Publishing from the University of Michigan (2001), (retrieved August 10, 2004)
  2. 2.
    Callan, J., Connell, M.: Query-Based Sampling of Text Databases. ACM Transactions on Information Systems 19(2), 97–130 (2001)CrossRefGoogle Scholar
  3. 3.
    Fravano, L., Change, K., Garcia-Molina, H., Paepcke, A.: STARTS Stanford Proposal for Internet Meta-Searching. In: Proceedings of the ACM-SIGMOD International Conference on Management of Data (1997)Google Scholar
  4. 4.
    Gravano, L., Ipeirotis, P.G., Sahami, M.: QProber: A System for Automatic Classification of Hidden-Web Databases. ACM Transactions on Information Systems (TOIS) 21(1) (2003)Google Scholar
  5. 5.
    Heß, M., Drobnik, O.: Clustering Specialised Web-databases by Exploiting Hyperlinks. In: Proceedings of the Second Asian Digital Library Conference (1999)Google Scholar
  6. 6.
    Hedley, Y.L., Younas, M., James, A., Sanderson, M.: Query-Related Data Extraction of Hidden Web Documents. In: Proceedings of SIGIR (2004)Google Scholar
  7. 7.
    Lin, K.I., Chen, H.: Automatic Information Discovery from the Invisible Web. In: International Conference on Information Technology: Coding and Computing (2002)Google Scholar
  8. 8.
    Meng, W., Wang, W., Sun, H., Yu, C.: Concept Hierarchy Based Text Database Categorization. International Journal on Knowledge and Information Systems 4(2), 132–150 (2002)CrossRefGoogle Scholar
  9. 9.
    Salton, G., McGill, M.: Introduction to Modern Information Retrieval. McCraw-Hill, New York (1983) Google Scholar
  10. 10.
    Sugiura, A., Etzioni, O.: Query Routing for Web Search Engines: Architecture and Experiment. In: 9th WWW Conference (2000)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Yih-Ling Hedley
    • 1
  • Muhammad Younas
    • 1
  • Anne James
    • 1
  • Mark Sanderson
    • 2
  1. 1.School of Mathematical and Information SciencesCoventry UniversityCoventryUK
  2. 2.Department of Information StudiesUniversity of SheffieldSheffieldUK

Personalised recommendations