Research on New Algorithm of Topic-Oriented Crawler and Duplicated Web Pages Detection

  • Yong-Heng Zhang
  • Feng Zhang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7390)


To improve the retrieval efficiency and performance of the large scale information retrieval systems, analyzed existing algorithm for Web search and duplicated Web pages detection. However, it has some drawback in terms of precision and efficiency because of its generality and no specialty. In this paper, with crawler and duplicated pages analysis, addressed two issues of the topic-oriented Web crawler and near-replicas detection. One is how to make the definition of the topic; the other is how to eliminate duplicate pages. It aimed to visit only topic-oriented pages, and got a great scale of hyperlinks which link to the topic-oriented pages. The crawl and Web pages detection method is a novel one, which was based on the semi-structured features of the website and content information. The results of experiment show that it is better than that of the existing algorithms proposed in the literature.


topic-oriented search engine Crawler near-replicas detection 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Yu, H.J., Liu, Y.Q., Zhang, M., Ru, L.Y., Ma, S.P.: Research in Search Engine User Behavior Based on Log Analysis. Journal of Chinese Information Processing 21(1), 109–114 (2007) (in Chinese with English abstract)Google Scholar
  2. 2.
    Bra, P.D., Houdben, G., Kornatzky, Y.: Information Retrieval in Distributed Hypertexts. In: Proc of the 4th RIAO Conference, New York, pp. 481– 491 (1994)Google Scholar
  3. 3.
    Gant, G., Srinivasan, P.: Topic-Driven Crawlers: Machine Learningissues. In: Proc of ACM TOIT (2004)Google Scholar
  4. 4.
    Angkawattanawit, N., Rungsawang, A.: Learnable Topic- Specific Web Crawler. Massive Information & Knowledge Engineering 28(2), 97–114 (2005)Google Scholar
  5. 5.
    Menczer, F., Pant, G., Srinivasan, P.: Topic Web Crawlers: Evaluating Adaptive Algorithm. ACM Transactions on Internet Technology 4(4), 378–419 (2004)CrossRefGoogle Scholar
  6. 6.
    Ye, S.Z., Ji, R.W., Ma, W.Y.: A Systematic Study on Parameter Correlations in Large-Scale Duplicate Document Detection. Knowledge and Information Systems 14, 217–232 (2007)CrossRefGoogle Scholar
  7. 7.
    Hosseini, S., Cox, J.R.: Optimal Solution of Off-line And On-Line Generalized Caching. Technical Report, WUCS-96-20, washington University at St. Louis (1996)Google Scholar
  8. 8.
    Voorhees, E.M.: The Philosophy of Information Retrieval Evaluation. In: Peters, C., Braschler, M., Gonzalo, J., Kluck, M. (eds.) CLEF 2001. LNCS, vol. 2406, pp. 355–370. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  9. 9.
    Scholer, F., Williams, H.E., Yiannis, J., Zobel, J.: Compression of Inverted Indexes for Fast Query Evaluation. In: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 222–229 (2002)Google Scholar
  10. 10.
    Cockburn, A., Jones, S.: Which way now? Analysing and easing inadequacies in WWW navigation. International Journal of Human-Computer Studies 45, 105–129Google Scholar
  11. 11.
    Silverstein, C., Henzinger, M., Marais, H.: Analysis of A Very Large Web Search Engine Query Log. SIGIR Forum 33(1), 6–12 (1998)CrossRefGoogle Scholar
  12. 12.
    Andrei, B.: A Taxonomy of Web Search. SIGIR Forum 36(2) (2002)Google Scholar
  13. 13.
    Voorhees, E.M., Lori, P., Buckland, E.: The Eleventh Text Retrieval Conference (TREC 2002), vol. 11. National Institute of Standards and Technology, NIST (2003)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Yong-Heng Zhang
    • 1
  • Feng Zhang
    • 1
    • 2
  1. 1.School of Information EngineeringYulin UniversityYulinChina
  2. 2.School of automationNorthwestern Polytechnical UniversityXi’anChina

Personalised recommendations