Enhanced Web Crawler Design to Classify Web Documents Using Contextual Metadata

  • L. Rajesh
  • V. Shanthi
  • V. Varadhan
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 336)


World Wide Web (WWW) is a biggest place of information repository in the Universe. A common man often seeks the assistance of Web for gathering information to enrich and enhance the knowledge of his interest to become an expert in his/her field. More often than not, search engines come in handy to provide information to the user. The nightmare of the search engines relies on the relevancy of the result-set presented to the user. To provide more relevant results, most of the search engines will have Web crawler in its armory as an important component to index the Web pages. Web crawlers (also called Web Spiders or Robots) are programs used to download documents from the internet. A focused crawler is a specialized crawler which will search for and index the Web page of a particular topic, thus reducing the amount of network traffic and download. This paper determines and identified a set of factors to determine the relevancy of Web documents and introduces a Contextual metadata framework to summarize the captured relevancy data that can be used to categorize and sort results and in essence to improve the quality of the result-set presented to the end user.


Focused crawler Web popularity Link analysis Contextual metadata 


  1. 1.
    Lawrence, S., Giles, C.: Accessibility of information on the web. Comput. J. Nat. 400(6740), 107–109 (1999)CrossRefGoogle Scholar
  2. 2.
    Qin, J., Zhou, Y., Chau, M.: Building domain-specific web collections for scientific digital libraries: a meta search enhanced focused crawling method. Digital Libraries, 2004. Proceedings of the 2004 Joint ACM/IEEE Conference on. IEEE (2004)Google Scholar
  3. 3.
    Liu, B.: Web data mining (from Chaps. 6, 7, 8). pp. 183–235, 237–270, 273–31. Springer, Berlin (2007)Google Scholar
  4. 4.
    Cho, J., Garcia-Molina, H., Page, L.: Efficient crawling through URL ordering. Comp. Netw. ISDN Syst. 30(1–7):161–172 (1998)Google Scholar
  5. 5.
    Shkapenyuk, V., Suel, T.: Design and implementation of a high-performance distributed web crawler. Data Engineering, 2002. Proceedings. 18th International Conference on. IEEE (2002)Google Scholar
  6. 6.
    Boldi, P., Codenotti, B., Santini, M., Vigna, S.: Ubicrawler: a scalable fully distributed web crawler (2002)Google Scholar
  7. 7.
    Yuan, X., Harms, J.: An efficient scheme to remove crawler traffic from the internet. In: Proceedings of the 11th International Conferences On Computer Communications and Networks, pp. 90–95. IEEE, New York (2002)Google Scholar
  8. 8.
    Chakrabarti, S.: Mining the Web (from Chaps. 2, 3). Giga-Pedia, pp. 17–77Google Scholar
  9. 9.
    Chakrabarti, S., van den Berg, M., Dom, B.: Focused crawling: a new approach to topic-specific web resource discovery. Comput. Netw. 31(11), 1623–1640 (1999)Google Scholar
  10. 10.
    Faniel, I.M., Yakel, E.,: Significant properties as contextual metadata. J. Libr. Metadata 11(3–4), 155–165Google Scholar

Copyright information

© Springer India 2015

Authors and Affiliations

  1. 1.Department of CSASCSVMV UniversityKanchipuramIndia
  2. 2.Department of MCASt. Joseph’s College of EngineeringChennaiIndia
  3. 3.Senior Technology ConsultantBangaloreIndia

Personalised recommendations