World Wide Web

, Volume 19, Issue 4, pp 605–631 | Cite as

Focused crawling for the hidden web

  • Panagiotis Liakos
  • Alexandros Ntoulas
  • Alexandros Labrinidis
  • Alex Delis


A constantly growing amount of high-quality information resides in databases and is guarded behind forms that users fill out and submit. The Hidden Web comprises all these information sources that conventional web crawlers are incapable of discovering. In order to excavate and make available meaningful data from the Hidden Web, previous work has focused on developing query generation techniques that aim at downloading all the content of a given Hidden Web site with the minimum cost. However, there are circumstances where only a specific part of such a site might be of interest. For example, a politics portal should not have to waste bandwidth or processing power to retrieve sports articles just because they are residing in databases also containing documents relevant to politics. In cases like this one, we need to make the best use of our resources in downloading only the portion of the Hidden Web site that we are interested in. We investigate how we can build a focused Hidden Web crawler that can autonomously extract topic-specific pages from the Hidden Web by searching only the subset that is related to the corresponding area. In this regard, we present an approach that progresses iteratively and analyzes the returned results in order to extract terms that capture the essence of the topic we are interested in. We propose a number of different crawling policies and we experimentally evaluate them with data from four popular sites. Our approach is able to download most of the content in search in all cases, using a significantly smaller number of queries compared to existing approaches.


Hidden Web Focused Crawling Topic-sensitive Query selection 



This work has been partially supported by SocWeb and Sucre FP7 EU projects. A preliminary version of the work appeared in the Proc. of the 13th Int. Conf. on Web Information Systems Engineering [15].


  1. 1.
    Álvarez, M., Raposo, J., Pan, A., Cacheda, F., Bellas, F., Carneiro, V.: Deepbot: A focused crawler for accessing hidden web content. In: Proceedings of the 3rd International Workshop on Data Enginering Issues in E-commerce and Services (EC), pp. 18–25, San Diego (2007)Google Scholar
  2. 2.
    Barbosa, L., Freire, J.: Siphoning hidden-web data through keyword-based interfaces. In: SBBD, pp. 309–321. Distrito Federal, Brasil (2004)Google Scholar
  3. 3.
    Barbosa, L., Freire, J.: Searching for hidden-web databases. In: Proceedings of the 8th International WebDB, pp. 1–6, Baltimore (2005)Google Scholar
  4. 4.
    Barbosa, L., Freire, J.: An adaptive crawler for locating hidden-web entry points. In: Proceedings of the 16th International Conference on World Wide Web (WWW), pp. 441–450. Banff, Canada (2007)Google Scholar
  5. 5.
    Bergholz, A., Chidlovskii, B.: Crawling for domain-specific hidden web resources. In: Proceedings of the 4th International Conference on Web Information Systems Engineering (WISE), pp. 125–133, Roma (2003)Google Scholar
  6. 6.
    Bergman, M.K.: The deep web. surfacing hidden value. J. Electron. Publ. 7(1), 1–17 (2001)CrossRefGoogle Scholar
  7. 7.
    Cafarella, M.J., Madhavan, J., Halevy, A.: Web-scale extraction of structured data. SIGMOD Rec. 37(4), 55–61 (2009)CrossRefGoogle Scholar
  8. 8.
    Chakrabarti, S., van den Berg, M., Dom, B.: Focused crawling: A new approach to topic-specific web resource discovery. In: In Proceedings of the 8th International Conference on World Wide Web (WWW), pp. 1623–1640, Toronto (1999)Google Scholar
  9. 9.
    Diligenti, M., Coetzee, F., Lawrence, S., Giles, C.L., Gori, M.: Focused crawling using context graphs. In: Proceedings of the 26th International Conference on Very Large Data Bases (VLDB), pp. 527–534, Cairo (2000)Google Scholar
  10. 10.
    Friedman, N., Geiger, D., Goldszmidt, M.: Bayesian network classifiers. Mach. Learn. 29(2–3), 131–163 (1997)CrossRefzbMATHGoogle Scholar
  11. 11.
    Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The weka data mining software: An update. SIGKDD Explor. Newsl. 11(1), 10–18 (2009)CrossRefGoogle Scholar
  12. 12.
    He, B., Patel, M., Zhang, Z., Chang, K.C.-C.: Accessing the deep web: A survey. Communications of the ACM 50(5), 94–101 (2007)Google Scholar
  13. 13.
    Ipeirotis P.G., Gravano, L.: Distributed search over the hidden web: Hierarchical database sampling and selection. In: Proceedings of the 28th International Conference on Very Large Data Bases (VLDB), pp. 394–405, Hong Kong (2002)Google Scholar
  14. 14.
    Ipeirotis, P.G., Gravano, L., Sahami, M.: Probe, count, and classify: Categorizing hidden web databases. SIGMOD Rec. 30, 67–78 (2001)CrossRefGoogle Scholar
  15. 15.
    Liakos P., Ntoulas, A.: Topic-sensitive hidden-web crawling. In: Proceedings of the 13th International Conference on Web Information Systems Engineering (WISE), pp. 538–551, Paphos (2012)Google Scholar
  16. 16.
    Lim, T.-S., Loh, W.-Y., Shih, Y.-S.: A comparison of prediction accuracy, complexity, and training time of old, thirty-three algorithms, new classification. Mach. Learn. 40(3), 203–228 (2000)CrossRefzbMATHGoogle Scholar
  17. 17.
    Lu, J., Wang, Y., Liang, J., Chen, J., Liu, J.: An approach to deep web crawling by sampling. In: Proceedings of the 2008 IEEE / WIC / ACM International Conference on Web Intelligence, (WI), pp. 718–724, New SouthWales (2008)Google Scholar
  18. 18.
    Madhavan, J., Ko, D., Kot, Ł., Ganapathy, V., Rasmussen, A., Halevy, A.: Google’s deep web crawl. Proc. VLDB Endow. 1(2), 1241–1252 (2008)CrossRefGoogle Scholar
  19. 19.
    McCandless, M., Hatcher, E., Gospodnetic, O.: Lucene in Action, 2nd. Manning Publications Co., Greenwich (2010)Google Scholar
  20. 20.
    Noh, S., Choi, Y., Seo, H., Choi, K., Jung, G.: An intelligent topic-specific crawler using degree of relevance. In: IDEAL, volume 3177 of Lecture Notes in Computer Science, pp. 491–498 (2004)Google Scholar
  21. 21.
    Ntoulas, A., Zerfos, P., Cho, J.: Downloading textual hidden web content through keyword queries. In: Proceedings of the 5th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL), pp. 100–109, Denver (2005)Google Scholar
  22. 22.
    Platt, J.C.: Advances in Kernel Methods. Chapter Fast Training of Support Vector Machines Using Sequential Minimal Optimization, pp. 185–208. MIT Press, Cambridge (1999)Google Scholar
  23. 23.
    Raghavan, S., Garcia-Molina, H.: Crawling the hidden web. In: Proceedings of the 27th International Conference on Very Large Data Bases (VLDB), p. 2001, RomaGoogle Scholar
  24. 24.
    Salton, G., McGill, M.J.: Introduction to Modern Information Retrieval. McGraw-Hill Inc., New York (1986)zbMATHGoogle Scholar
  25. 25.
    Schonhofen, P.: Identifying document topics using the wikipedia category network. In: Proceedings of the 2006 IEEE/WIC/ACM International Conference on Web Intelligence (WI), pp. 456–462, Hong Kong (2006)Google Scholar
  26. 26.
    Wang, Y., Lu, J., Chen, J.: Crawling deep web using a new set covering algorithm. In: Proceedings of the 5th International Conference on Advanced Data Mining and Applications (ADMA), pp. 326–337, Beijing (2009)Google Scholar
  27. 27.
    Wu, P., Wen, J.-R., Liu, H., Ma, W.-Y. : Query selection techniques for efficient crawling of structured web sources, p. 47, Atlanta (2006)Google Scholar
  28. 28.
    Wu, W., Yu, C., Doan, A., Meng, W.: An interactive clustering-based approach to integrating source query interfaces on the deep web. In: Proceedings of the 2004 ACM SIGMOD International Conference on Management of Data, pp. 95–106, Paris (2004)Google Scholar
  29. 29.
    Yang, Y., Bansal, N., Dakka, W., Ipeirotis, P., Koudas, N., Papadias D.: Query by document. In: Proceedings of the 2nd ACM International Conference on Web Search and Data Mining (WSDM), pp. 34–43, Barcelona (2009)Google Scholar
  30. 30.
    Zhang, Z, He, B., Chang, K. C.-C.: Understanding web query interfaces: Best-effort parsing with hidden syntax. In: Proceedings of the 2004 ACM SIGMOD International Conference on Management of Data, pp. 107–118, Paris (2004)Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Panagiotis Liakos
    • 1
  • Alexandros Ntoulas
    • 1
    • 2
  • Alexandros Labrinidis
    • 3
  • Alex Delis
    • 1
  1. 1.Universtiy of AthensAthensGreece
  2. 2.ZyngaSan FransiscoUSA
  3. 3.University of PittsburghPittsburghUSA

Personalised recommendations