Research on New Algorithm of Topic-Oriented Crawler and Duplicated Web Pages Detection
To improve the retrieval efficiency and performance of the large scale information retrieval systems, analyzed existing algorithm for Web search and duplicated Web pages detection. However, it has some drawback in terms of precision and efficiency because of its generality and no specialty. In this paper, with crawler and duplicated pages analysis, addressed two issues of the topic-oriented Web crawler and near-replicas detection. One is how to make the definition of the topic; the other is how to eliminate duplicate pages. It aimed to visit only topic-oriented pages, and got a great scale of hyperlinks which link to the topic-oriented pages. The crawl and Web pages detection method is a novel one, which was based on the semi-structured features of the website and content information. The results of experiment show that it is better than that of the existing algorithms proposed in the literature.
Keywordstopic-oriented search engine Crawler near-replicas detection
Unable to display preview. Download preview PDF.
- 1.Yu, H.J., Liu, Y.Q., Zhang, M., Ru, L.Y., Ma, S.P.: Research in Search Engine User Behavior Based on Log Analysis. Journal of Chinese Information Processing 21(1), 109–114 (2007) (in Chinese with English abstract)Google Scholar
- 2.Bra, P.D., Houdben, G., Kornatzky, Y.: Information Retrieval in Distributed Hypertexts. In: Proc of the 4th RIAO Conference, New York, pp. 481– 491 (1994)Google Scholar
- 3.Gant, G., Srinivasan, P.: Topic-Driven Crawlers: Machine Learningissues. In: Proc of ACM TOIT (2004)Google Scholar
- 4.Angkawattanawit, N., Rungsawang, A.: Learnable Topic- Specific Web Crawler. Massive Information & Knowledge Engineering 28(2), 97–114 (2005)Google Scholar
- 7.Hosseini, S., Cox, J.R.: Optimal Solution of Off-line And On-Line Generalized Caching. Technical Report, WUCS-96-20, washington University at St. Louis (1996)Google Scholar
- 9.Scholer, F., Williams, H.E., Yiannis, J., Zobel, J.: Compression of Inverted Indexes for Fast Query Evaluation. In: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 222–229 (2002)Google Scholar
- 10.Cockburn, A., Jones, S.: Which way now? Analysing and easing inadequacies in WWW navigation. International Journal of Human-Computer Studies 45, 105–129Google Scholar
- 12.Andrei, B.: A Taxonomy of Web Search. SIGIR Forum 36(2) (2002)Google Scholar
- 13.Voorhees, E.M., Lori, P., Buckland, E.: The Eleventh Text Retrieval Conference (TREC 2002), vol. 11. National Institute of Standards and Technology, NIST (2003)Google Scholar