Advertisement

Parallel Web Spiders for Cooperative Information Gathering

  • Jiewen Luo
  • Zhongzhi Shi
  • Maoguang Wang
  • Wei Wang
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3795)

Abstract

Web spider is a widely used approach to obtain information for search engines. As the size of the Web grows, it becomes a natural choice to parallelize the spider’s crawling process. This paper presents a parallel web spider model based on multi-agent system for cooperative information gathering. It uses the dynamic assignment mechanism to wipe off redundant web pages caused by parallelization. Experiments show that the parallel spider is effective to improve the information gathering performance within an acceptable interaction efficiency cost for controlling. This approach provides a novel perspective for the next generation advanced search engine.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Chau, J.M.: Design and evaluation of a multi-agent collaborative. Web mining system Decision Support Systems 35, 167–183 (2003) Google Scholar
  2. 2.
    Cho, J., Garcia-Molina, H.: Parallel Crawlers. In: Proc. of the 11th International World–Wide Web Conference Google Scholar
  3. 3.
    Shi, Z.: MAGE: An Agent-Oriented Programming Environment. In: Third IEEE International Conference on Cognitive Informatics (ICCI 2004), pp. 250-257Google Scholar
  4. 4.
    Zhuge, H.: China’s E-Science Knowledge Grid Environment. IEEE Intelligent Systems 19(1), 13–17 (2004) Google Scholar
  5. 5.
    Nodine, M., Fowler, J., Ksiezyk, T., Perry, B., Taylor, M., Unruh, A.: Active Information Gathering in InfoSleuth. In International Journal of Cooperative Information Systems 9(1/2), 3–28 (2000) Google Scholar
  6. 6.
    Dong, M., Liu, S., Zhang, H., Shi, Z.: Parallel Web Spider Based on Intelligent Agent. In: Proceedings of The 5th Pacific Rim International Workshop on Multi-Agents (PRIMA 2002), Tokyo (2002) Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Jiewen Luo
    • 1
    • 2
  • Zhongzhi Shi
    • 1
  • Maoguang Wang
    • 1
    • 2
  • Wei Wang
    • 2
  1. 1.The Key Laboratory of Intelligent Information Processing, Institute of Computing TechnologyChinese Academy of SciencesBeijingChina
  2. 2.Graduate School of Chinese Academy of Sciences 

Personalised recommendations